credibility study

7/29/2019 Credibility Study

1/50

Research Study:

A pilot analysis of credibility of pro-Israel messages

Israel-Arab Studies Program

Jerusalem Center for Public Affairs

March 1, 2013

The following study is a pilot conducted by the students of the JCPAsIsrael-Arab Studies Program. As explained in the following presentation,the goals of the study included seeing how different statements typicallyused as pro-Israel messages are perceived in terms of credibility orbelievability. We would expect that certain sources would be seen asmore credible than others, and vice-versa.

In selecting the current sample for study, we are cognizant that they didnot represent a normal population. Our subjects were all traditionallyobservant Jewish students and all could reliably be described as

staunchly pro-Israel in their overall outlook. This makes the resultsfound even more interesting, as a significant portion of our sample castdoubt on the credibility of traditional pro-Israel sources of informationwhile, at the same time, seeing non-affiliated sources who would presentthe very same information as far more credible.

As the accompanying analysis and explanation will show, the samplewas far from monolithic. While some showed very standard responsesto be expected from such a pro-Israel group, a significant portionconsistently assigned relatively low ratings of credibility to pro-Israelsources, especially when rating the believability of messages presented

by AIPAC, the pro-Israel lobby in the USA. Some students showed clearbias against non-affiliated sources, as they assigned very credibleratings to pro-Israel sources while at the same time assigning a muchlower rating to a non-affiliated source, even though the very samestatement was being evaluated. For a significant portion of students, theopposite held true and significantly higher ratings of credibility wereassigned to non-affiliated rather than pro-Israel sources.

The significance of these findings is in the fact that these resultsshowed about half of this sample of traditional pro-Israel students notshowing inordinate faith in the messages of pro-Israel sources. Whilemost of the statistical analysis yielded significant results, this wouldhave been even more pronounced had we used less rigorous 1-tailedanalysis, which in this population sample, would have been appropriate.

We would want to replicate this study in other, less identifiably pro-Israel populations, but we suspect that the results obtained here wouldbe even more pronounced in those groups.

Irwin J (Yitzchak) Mansdorf, PhD


2/50

How Israels policy positions and behavior is perceived by the international

community has long been a topic of much discussion. Many organizations

within and outside of Israel deal with the issue of Israels image. To this end,

campaigns and activities that range from campus activism to social media

presence to public advertising are used to present a variety of positions and

often to defend Israel against accusations of abuse towards the Palestinian

Arab population.

This study looks at how information is perceived by others. Specifically, we will

be looking at information related to the Israel-Palestinian dynamic and how

that information relates to where it originated from. We are interested in

learning whether there is a relationship between the credibility, or believability

of a statement and the source of that statement, i.e., who made it.


3/50

We hypothesize that there may be relationship between how one evaluates a

particular statement and where that statement originated from. This

relationship will be different for different populations. While some populations

may consider a particular source quite credible, others may look at that very

same source with skepticism.

In preparing and presenting policy positions directed at particular populations,

understanding the characteristics of what constitutes a reliable and believable

source of information is critical to having that information assimilated.


4/50

Our basic goal is to evaluate certain statements made by official and NGO

sources and see how these statements are perceived, depending on the

source.

Is there a difference in believability of a statement, depending on who is

presenting it? How do people perceive messages emanating from the Israeli

government? From pro-Israel organizations? From the media?

The results may have important ramifications for public diplomacy.


5/50

Our first group for study is a group thought to be rather monolithic.

We looked at the attitudes of 18-19 year old foreign students (mostly USA

based but also from UK, Australia and South Africa) studying in religiousinstitutions in Jerusalem. These students have strong Jewish identities and are

overwhelmingly considered pro-Israel in their political outlooks.

Their study in Jerusalem is in the context of a post-high year of intense Jewish

studies prior to commencing university studies. The majority of these students

will be returning to their home countries at the end of the academic year.

As a rule, this group would not be considered representative of a generalstudent population. It would be reasonable to say that their views would be

significantly more pro-Israel than other student groups.


6/50

This pilot exploration involved a total of 88 students (35M, 53F) who were

randomly approached to participate in the study.

We used ten typical pro-Israel statements or messages and gauged thebelievability (credibility) of each statement, depending on the source of the

statement. We presented the statement and asked how credible it would be if

it originated from one of five sources: The Israeli government (MFA), the pro-

Israel USA lobby (AIPAC), a media source (CBS News), a Jewish-Muslim

student association and a human rights organization.

Ratings were recorded on a hand-out sheet which contained the ten

statements and a rating scale.

The expectation was that there would be an interaction between the

information contained in the statement and the source of the information.

Certain statements may appear credible notwithstanding the source, while

others may vary depending on the source.


7/50

Each student was first explained the nature of the study and presented with an

example of the rating would take place.


8/50

Ten statements were presented and listed above.

These are the first 5 statements. They represent common pro-Israel

messages typically promoted by organizations involved in Israel advocacy.


9/50

These are statements 5-10.


10/50 1

The rating scale was a 10-point scale accompanied by a verbal legend that

scaled credibility from not credible to very credible.


11/50

These are the five sources of information that each student was to consider,

one at a time, in rating the credibility of the information presented in the

statement.


12/50 1

Our initial analysis looked at the gross scores of group. We averaged the

ratings of each individual statement and computed the average credibility

score for each individual statement and source for the group as a whole.

We also looked at how each individual student rated the statements as a

whole, averaging their ratings for the statements across all sources of

information.

We also looked at the percentages of students who ranked the statements

toward the lower end of credibility, i.e., no more than somewhat credible.


13/50 1

Our results showed that the mean rating level across the group was rather

moderate, with overall ratings trending towards the middle of the scale

(somewhat credible range).

However, this average does not take into consideration broad groups of

students whose ratings could be clustered into discrete categories. While

certain students appeared to tend to rate the statements (allstatements)

generally high, no matter what the source was, others tended show different

evaluations for different sources of information. Thus, some students seemed

to accept the credibility of the statement as is, while others saw it varying

according to who presented the information.

As seen in statistical analysis, there was a statistically significant differenceover all of the questions between at least one of the pro-Israel sources and

the non-affiliated sources, with the pro-Israel sources consistently seen as

having less credibility (no more than somewhat credible).


14/50 1

The above is a graph that presents each individual students average rating for

all statements.

While a few students show extremely high averages, the majority show scoresthat cluster toward the middle range of the scale.


15/50 1

This graph presents the data broken into various credibility levels. On the 10-

point scale, scores of 1 or 2 were considered not very credible, scores of 3-4

were considered barely credible, 5-6 somewhat credible, 7-8 mostly

credible and scores of 9-10 were considered very credible.

The graph shows a pattern very close to a statistically normal distribution,

with the majority of scores in the middle, in the somewhat credible range,

with a somewhat higher trend in the mostly credible range than in the barely

credible range.


16/50 1

Breaking this pattern info percentages shows that while over 32% of the group

felt that most of the statements were mostly or very credible, a considerable

amount, over 14%, felt the statements to be either not very or barely

credible.

The majority, over 53%, scored the overall credibility level of all the

statements, as somewhat credible.

Taken together, over 68% of the sample did not see the overall credibility of

the statements from the sources presented as more than somewhat credible.


17/50 1

While the overall averages and distribution of percentages provide a broad

overview of the results, the individual items surveyed present a more

comprehensive and more interesting level of analysis. It is here that we see

how the actual source of the information impacts on the credibility of the

statements.

We would expect certain statements to be credible no matter what the source

is. Our results, however, show this not to be the case with all the students

surveyed.

While the results differed somewhat depending on the statement tested, there

is a clear trend which shows the students considering certain sources more

credible than others. Formal statistical analysis showed that most of thecomparisons between credibility of the pro-Israel and non-affiliated sources

showed significant differences, with lower credibility ratings for the pro-Israel

sources.


18/50 1

This is the first statement presented.


19/50 1

Overall, for the group as a while, the statement trended towards the mid-high

range of somewhat credible.


20/50 2

The above table presents the percentages of students scoring the statement

as either not very, barely or somewhat credible, depending on the source.

This would be a score of 5 or less and represents the lower end of credibility of

the scale.

A further analysis looking at each source of information shows a trend where

AIPAC is given the lowest credibility score, with over 46% of the students

rating their credibility for the statement on compromises for peace as no

more than somewhat credible.

The highest credibility ratings were given to the Jewish-Muslim Student

Alliance.

The t statistic for the difference between AIPAC and JMSA was 2.14, yielding

a statistically significant difference between the percentages at the p


21/50 2

This is the second statement presented to the students.


22/50 2

The statement The IDF is the most moral army in the world elicited the

following average ratings.

The Citizens Association for Human Rights appeared to have a noticeablyhigher level of credibility that the other sources, with the MFA and AIPAC

showing the lowest levels of credibility.


23/50 2

A percentage analysis of below 5 scores shows a more dramatic picture.

Over half of the students surveyed considered the statement, when made by

either the MFA or AIPAC, to have limited credibility. On the other hand, about85% of the students saw CBS news as a generally credible source for this

statement.

The t statistic for the difference between AIPAC or the MFA and any of the

other sources yielded statistically significant differences between the

percentages at the p


24/50 2

This is the third statement presented to the students.


25/50 2

Average scores show a similar trend to the other statements, white generally

higher scores for sources of information not identifiable as distinctly pro-

Israel.


26/50 2

A similar trend is seen in the in-depth analysis of the sources of information

with regards to credibility.

AIPAC appears to be perceived in this statement as having the lowest level ofcredibility. The higher levels of credibility are found with the non-affiliated or

non-pro-Israel identified sources.

The t statistic for the difference between AIPAC and any of the other non-

affiliated sources yielded statistically significant differences between the

percentages at the p


27/50 2

This was the fourth statement tested.


28/50 2

Raw average scores appear to show a lower credibility level for AIPAC and a

higher level for the non- pro-Israel sources.


29/50 2

A similar trend is seen in the in-depth analysis of the sources of information

with regards to credibility.

AIPAC appears to be perceived in this statement as having the lowest level ofcredibility. Over 67% of those surveyed found AIPAC to be no better than a

somewhat credible source of information with regards to this statement. Only

about 26% of the group saw the non-affiliated human rights group as having

limited credibility.

Statistical significance at the p=< .05 level was seen between the AIPAC

scores and all other non-affiliated groups. Statistical significance was also

seen between the MFA percentages and all other non-affiliated groups except

for CBS news.


30/50 3

The issue of apartheid was presented in the fifth statement tested.


31/50 3

A noticeable difference between the pro-Israel and the non-affiliated sources

on the issue of apartheid is seen, with the pro-Israel sources showing only

limited credibility while the non-affiliated sources showing credibility levels

approaching or beyond mostly credible levels.


32/50 3

The analysis of individual student ratings shows a more dramatic split. Over

half the students rated the pro-Israel sources as no more than somewhat

credible., trending downwards. On the other hand, over 88% saw the human

rights group on this issue as being at leastsomewhat credible, trending

towards higher ratings.

These results yielded statistically significant differences (p=


33/50 3

The sixth statement tested referenced the rights of Arab citizens within Israel.


34/50 3

The raw scores show quite strong splits between the average rating of pro-

Israel sources and those of the non-affiliated sources, with higher credibility

scores for the non-affiliated sources.


35/50 3

Again, this strong split is seen in the number of students who rated openly

pro-Israel sources as at least somewhat credible. We again see AIPAC as

being viewed as the least credible source overall, with over half the

respondents seeing them as no morethan somewhat credible trending

downwards, versus the human rights group, who were viewed as a source

that is at least somewhat credible (trending upwards) by about 82% of the

sample.

These results yielded statistically significant differences (p=


36/50 3

The seventh statement asked students to rate the credibility of the various

sources with regards to their view on the West Bank settlements.


37/50 3

Here, most of the sources were about even, with a score averaging around

somewhat credible. Again, AIPAC was considered the least credible source.


38/50 3

While the differences on this issue tend to be less pronounced than on

previous statements, the pattern of response remains the same. The pro-

Israel sources are seen as the least credible, with over 44% seeing AIPAC as

no more than somewhat credible.

Statistical analysis on this statement showed only one comparison that was

significant at the p=


39/50 3

The eighth statement revolved around a relatively obscure issue, that of

Palestinian Arab rights in eviction proceedings.


40/50 4

Results of raw score averages again show a trend of lower credibility for the

pro-Israel sources and much higher credibility for the non-affiliated sources.


41/50 4

Noticeable differences in percentages of lower credibility ratings were again

seen here. The MFA results were lower in credibility than all of the non-

affiliated sources, but were statistically significantly lower (p=


42/50 4

The ninth statement presented spoke about the assertion that the Palestinian

Arabs have repeatedly turned down what have been called generous Israeli

offers for peace.


43/50 4

The trend of lower average credibility ratings for the MFA and AIPAC versus

the non-affiliated groups continued here.


44/50 4

While lower credibility scores were seen between the MFA and the other non-

affiliated sources, these differences this not reach statistical significance.

Differences between AIPAC and the other non-affiliated groups also werenoticeable, with lower credibility assigned to AIPAC. This lower credibility rating

was statistically significant when compared to both the Jew-Muslim group and

the human rights group.


45/50 4

The tenth and final statement presented referred to the claim of Israel being

the only democracy in the Middle East.


46/50 4

A trend generally similar to that found on previous items was seen in the

average scores of credibility ratings for this statement.

MFA and AIPAC credibility was seen as lower than that of any of the non-affiliated sources.


47/50 4

Both the MFA and AIPAC had noticeably lower ratings of credibility than the

other, non-affiliated groups. These differences were all statistically significant

(p=


48/50 4

This survey of randomly selected pro-Israel students yielded results which

are somewhat unexpected.

One might expect that these students, generally Orthodox, and generally fromeducational systems where Israel is seen in a very positive light, would see

information coming from pro-Israel sources as fairly credible.

While a sub-group responded in just this manner (with some students

appearing to blindly assign high ratings to all statements regarding of the

source), a (statistically) significant portion did not.


49/50 4

We can analyze results in a number of ways, but we see that the approach

where like-minded people sit around a table and decide what makes sense to

them may not yield effective intervention. Very often data-driven research

shows counterintuitive approaches to be more valid.

Results here show that intuitive thinking about what is effective in public

diplomacy has its limitations, as conventional wisdom would have expected

different levels of confidence in credibility that our results demonstrated.

This may be due to a number of factors. The research question created here

would be to understand what factors in a particular source of information

determine how a statement is ultimately viewed. Is it an organic attribute of

the source itself or is it a function of behaviors related to the source that

influences perception?

Our results seem to indicate that perceived impartiality or perceived fairness

may be related to perceived credibility. Can a previously not credible source

become more credible by doing something to change that perceptionor is

the characteristic immutable because of the nature of the source itself? Further

study into this question is critical to developing effective public diplomacy

approaches in the future.


50/50

As in many studies, there are limitations in this research.

First, we need to ask if the validity of the responses we found is intact. The

study was conducted by student volunteers not trained in research proceduresand the possibility of errant procedures in soliciting answers exist.

It is possible that the student volunteers did not adhere fully to acceptedresearch procedures. While the study was blind insofar as any bias towardsany particular type of response, we did not control for any individual studentvolunteer bias that may have existed.

It is also possible that the student responders did not fully understand thepurpose or intent of the inquiry. In general, we need to ask if the data gatheredis indeed valid or reliable.

We also need to ask if the sample is indeed representative of the populationwe selected to study. As a convenience sample, the possibility exists that thestudents asked to participate represent a biased sample.

Limitations in any study need to be considered in any analysis. While data may

credibility study

Documents