reality reporting and moderation apps …cse.iitd...reality reporting and moderation apps for...

R E A L I T Y R E P O RT I N G A N D M O D E R AT I O N A P P S F O RC O M M U N I T Y R E P O RT E R S I N R U R A L A R E A S

A dissertation submitted in partial fulfilment ofThe requirements for the degree of

Dual Degree(B.Tech + M.Tech)in

Computer Science and Engineering

Bymridu atray

Department of Computer ScienceIndian Institute of Technology

July 2013

C E RT I F I C AT E

This is to certify that the dissertation titled "Reality Reporting andModeration Apps for Community Reporters in Rural Areas“ beingsubmitted by Mridu Atray, Entry Number 2008CS50217 to the IndianInstitute of Technology Delhi, for the award of the degree of DualDegree(B.Tech+M.Tech) in Computer Science and Engineering, De-partment of Computer Science and Engineering, Indian Institute ofTechnology Delhi, is a record of bona fide research work carried outby her under our guidance and supervision at the Department ofComputer Science and Engineering, Indian Institute of Technology,Delhi. The work presented in this thesis has not been submitted else-where, either in part or full, for the award of any other degree ordiploma.

New Delhi, July 18, 2013

Dr. Aaditeshwar SethDepartment of Computer Science and EngineeringIndian Institute of Technology DelhiNew Delhi 110 016

A B S T R A C T

Citizen-sourced journalism media raise the need of moderation be-fore items can be made available to the public. This need for modera-tion acts as a bottleneck when we try to scale such platforms. An in-crease in the number of moderators appears a simple straightforwardsolution but needs investment in terms of the salaries and training ofthese moderators. In this study, we explore a hierarchical modera-tion model to meet the pressing need of moderation, hence reducingload on the main team of moderators and the time for an item to getpublished. The moderation is now done at two levels, one by certaincommunity representatives with the help of a mobile app. Respon-sibilities are split across the two levels in an effort to maximise thereduction of load at the central level while taking into account themobile handling skills of the community representatives.

In the current day, news comes in from a number of varied sources.Due to this there is increasing importance of presenting the user witha comprehensive digest of the latest developments. This had led tonew fields such as Data Journalism and Computational Journalismwhere increasing emphasis is paid on making available more thanwhat meets the eye normally. As the number of stories increase it isincreasingly important therefore to be able to recognize related itemsand develop visual interfaces which help in better communicatingthe news. Therefore, in this study, we also explore the idea of realityreporting. Under this, we aim to develop a means to create certain ac-tive issues from the reported stories and hence link related stories asa part of the moderation process. New and interesting web interfacesare also designed in order to better represent insights obtained fromthese.

iv

A C K N O W L E D G E M E N T S

First and foremost, I would like to express my sincere gratitude tomy thesis advisor, Dr.Aaditeshwar Seth for his wisdom, knowledge,and insight that guided me during this study. Besides my advisor, Iwould like to thank Mr. Dinesh Kapoor, Ms. Aparna Moitra and Ms.Sayonee Chatterjee from the GramVaani for their consistent guidancewhile understanding the working of Jharkand Mobile Vaani and theprocess of moderation in a citizen-sourced journalism platform.

I am also grateful to Rakshit Agrawal for his help and support inthe development of the website and helping in formulation of the hi-erarchical moderation model based on the initial field study.

I would like to thank my parents for their love and support.

v

C O N T E N T S

1 introduction 1

1.1 Motivation and Description 1

1.2 Literature Review and Related Work 4

2 landscaping study 9

3 the hierarchical moderation mode 13

3.1 Moderation Data Analysis 13

3.1.1 Analysis and Results 14

3.2 Moderation through the Goonj App 14

3.2.1 The Goonj Mobile App 15

3.2.2 Distribution of Moderation Responsibilties 15

3.2.3 System Flow 16

3.2.4 The Moderation App 18

3.3 Tracking performance of community representatives 19

4 reality reporting 21

5 validation of the hierarchical model 27

5.1 Results and Insights 27

5.1.1 Accuracy 27

5.1.2 Ease of use of the app 33

5.1.3 Distribution of load 34

5.2 Observation, challenges and suggestions 35

5.2.1 Observations and Conclusions 35

5.2.2 Challenges and Suggestions 36

6 website 39

6.1 Technical Specifications 39

6.2 Details of the Web Pages 40

6.3 Populating the Database 42

bibliography 45

vii

L I S T O F F I G U R E S

Figure 1 Distribution of responsibilities across levels. 16

Figure 2 Flow of data in the system. 17

Figure 3 Admin interface:Page showing list of storiesand their status 17

Figure 4 Screenshots of the Goonj app 18

Figure 5 Admin interface to manage and create new is-sues. 22

Figure 6 App Screen to assign topic, location, issue tags. 23

Figure 7 Interface for Issue Coverage with timeline. 24

Figure 8 Interface for Issue Coverage with map. 25

Figure 9 Visualisation of different topics covered at JMV 25

Figure 10 Accuracy in correctly identifying different cat-egories 29

Figure 11 Accuracy for crepB and crepC in different cat-egories 33

Figure 12 Main Page of the website. 39

Figure 13 Webpage for the topic channel - Education. 40

Figure 14 Webpage showing Reports and various statis-tics. 41

Figure 15 Step-Wise flow for the Python Script. 43

viii

L I S T O F TA B L E S

Table 1 Proportion of items in different categories. 14

Table 2 Moderation app test run 28

Table 3 Confusion Matrix among the various categories 29

Table 4 Confusion Matrix among for crepB 31

Table 5 Confusion Matrix for crepC 31

ix

1I N T R O D U C T I O N

1.1 motivation and description

Citizen journalism or participatory journalism has been gaining a lotof importance since the past decade. Citizen sourced journalism is animportant medium of expression for people, specially in rural India,who are often kept out of discussions regarding issues faced by them.There are different levels of participation by people in citizen journal-ism efforts[4]. This varies from audience participation in terms of pro-viding comments, writing persnal blogs to full-fledged participationwhere the media carries news solely from its users. With increasinglyimproved technology and the penetration of mobile phones, givingpeople lot of computation power in their hands, the medium of con-tribution of stories by the users is no longer limited to just text based.People can also now contribute audios, videos and photos to sup-port their stories. But the underlying fact remains that whatever thedegree of participation, and whatever the medium, there is a strongneed of screening the contributed items before they can be published.This is necessary in order to maintain the quality of the news of be-ing reported, the decisions being carried out and take into accountthe sensitivities of various parties.

The project is built around “Goonj: Media for Everyone”, an on-going project at GramVaani, a social technology company with theirhead office in Delhi. It is also more popularly known as JharkandMobile Radio (JMR) and has recently been renamed to Jharkand Mo-bile Vaani(JMV). It is a citizen radio-over-phone platform where peo-ple can place a missed call at a designated number and contributetheir stories. They can also call in at the same number to listen towhat other people have recorded. The items are published after mod-eration by a dedicated team. The medium of contribution of con-tent has been extended with the help of a Goonj App for smart-phones. These have been handed out to certain community represen-tatives/reporters. With the help of this application, the communityreporters can not only record audios (which can be done via basicfeature phones also by dialling the toll free number), but can alsoclick photographs and record videos. The app enables the commu-nity reporters to take interviews with concerned authorities/govern-ment officials and hence take the grievances of the people to wheresolutions can be found. Moreover, since this application is to be usedin areas where the internet connectivity is often not very good, it isbased on a data transfer framework designed specifically for flaky

1

2 introduction

internet connections. The audios, videos and photos collected by theGoonj app are sent to the main server in Delhi using this frameworkin a delay tolerant fashion.

As the number of stories contributed by people rises, the need formoderation can turn into a bottleneck for scaling such platforms. It isimportant to ensure that the items worth publishing are accessible byother people soon after they are recorded. The project aims to solvethis problem with the help of a scheme of hierarchical moderation asa solution. In this scheme, moderation is done at two levels: first bycommunity representatives with the help of a smartphone app andsecond, by a central team of dedicated moderators. Different respon-sibilities are split across the two levels, as explained later. This deci-sion is made to optimise the benefit of the heirarchical model, whilekeeping in mind the mobile handling capabilities of the communityrepresentatives who might not all be completely well educated.

Readers today in general face a growing volume of news and newssources. Hence there has been a recognised need for someone to makemore sense of all different items being reported and give the reader acomprehensive digest. There have been attempts to give the readera more comprehensive collection of stories from different sourcesby Google News[15], Techmeme[14], Breakingnews.com[16] and Flip-board[17]. These efforts are also taking into account the recent shifttowards the mobile paradigm of news consumption and are tailoringtheir interfaces to give the user maximum satisfaction.

Moreover information now comes in from a large number of var-ied sources.Also large information base has lead to the rise of newparadigms such as those of data journalism and computational jour-nalism. Computational journalism is considered to be the tool whichcan advance journalism by aiding in the way stories are discovered,aggregated and presented[1]. There is a need to develop new tech-nologies and algorithms which can help journalists uncover newerfacts and come up with news in public interest. Similarly there is alot of excitement currently in newsrooms for hiring “Data journalists”which can help make sense of the large amounts of data being col-lected by news organisations. Data journalism helps provide richerstories where instead of being the first ones to report about an event,one can stress more on what a particular development might actu-ally mean. This is where different infographics and data visualisationtechniques can also play a significant role. As the Data JournalismHandbook[3] explains:

“Gathering, filtering and visualizing what is happening beyondwhat the eye can see has a growing value”

In the context of JMV, other than the pressing need for scaling mod-eration, another problem that arises with an increasing number ofstories being reported is that the consumer is exposed to a large vol-ume of information. For a platform like JMV, there are items that are

1.1 motivation and description 3

recorded via the toll free number and related videos, audios, photosare also coming in from the Goonj app with the help of the com-munity representatives. Therefore there is a similar need to aggre-gate stories so that the people can gain more insight about the newsbeing reported as compared to the event-based reporting style. Weterm the aggregation of a set of related stories as an “issue”. This canbe follow-up on some news item reported previously at JMV, seeingthe progress of NREGA work over time, or some statistics/insightsgained from the various campaigns which are run at JMV. In thisproject, we also aim to develop suitable web interfaces for differentsuch issues, to be able to extract maximum information from a setof related stories and use interesting data visualisation techniquesfor the same. Not only does this provide more value to the generalconsumer of information at JMV, it also gives a more comprehensiveview of the current happenings related to an issue. This can also be ofgreat help in the newsroom to figure out the next steps in reportingthe issue. Data and information in such an organized form can thenalso be shown to various concerned authorities, hence making thevoices of the people to be heard where they need to be and improvechances of a resulting impact.

Along with the appropriate web interfaces, we also aim to add cer-tain features to the Goonj app by which the stories contributed bythe community reporter are better organized and hence aid him tofollow-up on the issue much more effectively.Currently all the com-munity reporter sees is a long list of audios/videos/photos taken byhim. Consider the scenario that community reporter X went to a vil-lage and recorded a video where the people reported that there wasno electricity in their village since the past week. X then goes to theconcerned official and records another audio/video with the officialwhere he promises to solve the problem within a certain time. NowX has to check after that time whether the official lived up to hispromise. It is difficult to keep track of the current status without anyorganized structure. The two stories till now are related and part ofthe same “issue", and hence should be seen together.

Along with added structure in terms of issues, Topic channels areintroduced to categorize different stories based on the broad top-ic/theme they belong to. These include things like Health, Education,and Governance among others. Forming dedicated channels basedon topics can help to tie up with different NGOs and other organi-zations working in that particular field to take better action on theissues and stories reported at Goonj. These organizations are calledChannel Partners. With this enhanced structure, one of the projectaims is to design and build a new website for better display of thestories. Currently a basic website to display the incoming stories isavailable at http://goonj.net .

To summarize, the project has the following three aims:

4 introduction

1. Design, build and validate a model to enable hierarchical mod-eration of the incoming stories as extension to the Goonj app.

2. Add features to the Goonj app to better organize stories col-lected via the app into issues and enable better tracking of thestatus of each.

3. Design new web interfaces to make story presentation more con-textual.

1.2 literature review and related work

The following papers were reviewed while developing the concept.

1. Emergent Practices Around CGNet Swara, A Voice Forum forCitizen Journalism in Rural India, ICTD’12 [6]: This paper talksabout the initiative of CGNet Swara, which is a project similar toJMR active in Chhatisgarh. The authors explain the deploymentof the system, and their experiences. It also delves into quali-tative and quantitative analysis of the data coming in, of thecallers, topic about which stories were reported among otherthings.

2. IVR Junction: Building Scalable and Distributed Voice Forumsin the Developing World, NSDR ’12 [7] : This paper discussesthe challenges and opportunities in creating scalable voice fo-rums. It recognizes that the need for moderation and also thefact that it can be a bottleneck in scaling such systems. The au-thors present a new open-source system, IVR Junction, whichleverages existing free services and commercial tools to simplifythe process of creating a voice forum.

3. Living Stories, Google Labs[13]: Living Stories is an experimen-tal project by Google in collaboration with The New York Timesand The Washington Post started in Dec 2009. This also aimedat presenting news in a more comprehensive manner by utilis-ing a timeline interface for depicting all the stories about onetopic had been featured on a single webpage. This helps thereader to understand what were all the different steps taken inregard to a particular news and hence be able to connect to thenews item better. This project was discontinued in Feb 2010.

4. Techmeme[14]: This is a project which helps its user to bettertrack the current trends in technology, hence called "Techmeme".This website is a news aggregator and curator which helps itsuser to get summarised reports of the news events in a day re-lated to technology on a single page, picking the stories fromvarious sources. Hence it gives its readers a one-stop to comeand gather all updates about technology related events. Just like

1.2 literature review and related work 5

readers here dont need to span different sources to get a hangof what’s happening, we also aim that the consumers of infor-mation from JMV can get a wholesome view of various storiesrelated to different issues which are reported at a single pageand not need to look for them himself.

5. Designing Mobile Interfaces for Novice and Low-Literacy Users,ACMTransactions on Computer-Human Interaction. 2011[5]: This studyexplores different interfaces for low-literacy and novice mobileusers. The authors conducted two studies comparing text-basedinterfaces to other different alternatives such as, one: automaticsolutions including graphics, spoken dialog and text-free userinterfaces and second: a live human operator. Based on thesestudies and interviews conducted with the subjects, the authorscite results regarding the comfort of novice users with the differ-ent mobile interface components. They also lay down certain de-sign recommendations while designing mobile user interfacesfor such users.

6. Comparing Semiliterate and Illiterate Users Ability to Transi-tion from Audio+Text to Text-Only Interaction, CHI’09[2]: Inthis paper, the authors establish fact that illiterate and semi-literate users can’t be both clubbed together into one categorywhen it comes to designing suitable user interfaces for them.This is so because for users with some basic literacy, who thoughmight not be able to read and write fluently, text provides an un-ambiguous mode of interaction. The authors conducted studieswhere they found that when semi-literate users were presentedwith an interface with both text and audio support, they soon re-duced their dependence on audio while no such improvementwas found in case of the fully illiterate users. The paper pro-vides interesting insights into the differences in the responsesof fully illiterate and semi-literate users to different UI compo-nents.

7. merinews[8] : merinews.com is a citizen journalism website run-ning actively in the country. Any citizen can register with thesite and submit news articles and photos. The articles are screenedand edited if necessary before being published. The photos aswell are screened before making them public. merinews alsomaintains a point system for its citizen journalists rewardingthe work done by them. People are also allow to comment onthe content on the website hence encouraging community dis-cussions over various topics of local and national interest. Thecomments are also well moderated maintaining quality of con-tent on the site. The site has also won various awards in theparticipatory news category.

6 introduction

8. GaurdianWitness[9]: GaurdianWitness is the user-generated con-tent wing of The Gaurdian, a British national daily newspa-per. People can contribute online and also through their mo-bile phones. GaurdianWitness is also available as a free app forthe Apple iPhone and Android phones with the help of whichpeople can contribute photos, videos and text. On the website,people can also explore contributions by others. Again there isa review process in place before publishing new content. Thereis a concept of “Assignments” where the editor decalres certaintopics where people can contribute, and people can send in newassignment ideas. On a more serious note, the participants alsoget to tie up their contributions to a breaking or fast-movingstory. Citizens are also free to report local news as stories onthe GaurdianWitness with the best stories being featured onThe Gaurdain website as well.

9. Computational Journalism, Communications of the ACM, 2011[1]:This is a contributed article explaining the how computer sci-ence can help journalists to come up with news with the publicinterest at the center. It talks about computational journalismand the role it can play to advance journalism by contribut-ing in easy and quick ways in topic detection, video analysis,personalization, aggregation, visualization,and sensemaking. Itstresses on the need of computer scientists and journalists towork together in order to keep public interest journalism upand running. It highlights the need for new specific algorithmsto be designed to help journalists process large volumes of pub-lically accessible records, to be able to match differenet datasetstogether to be able to draw insights which would not be possi-ble without the computational help.

10. The Data Journalism Handbook[3]: This handbook talks aboutwhat data journalism is and why it is important. It lays downvarious examples of data journalism across the world and alsotalks about various case studies regarding data journalism. Thehandbook itself features a very nice infographic about what allthe book talks about, hence in a way following what it preaches.it covers what happens inside the newsrooms where data jour-nalists are involved. They explore the same with the examplesof Australian Broadcasting Corporation, the BBC, the ChicagoTribune, the Guardian and the Zeit Online. It also talks in de-tail about how to acquire the required data, how to understandit and how to present the same to the targetted audience. Thiscovers various visualisation strategies and also explains when agood visualisation technique can benefit the audience to betterunderstand the stories being presented to them.

1.2 literature review and related work 7

11. USA Today Snapshots[10]: This is an example of usage of sim-ple inforgraphics for better representation of the stories. USAToday is an American national daily newspaper with one of thehighest circulation across the United States. USA Today Snap-shots post a new snapshot every weekday where they present avisually appealing infographic about new issues or new trends.There is also a “Quick Question” related to the snapshot whichreaders online can answer and hence helps in the collection ofmore interesting facts about the concerend issue or trend.

12. Message Machine, ProPublica[11]: ProPublica is an independent,non-profit newsroom established in New York, USA. They pri-marily work on investigative journalism in public interest. Awonderful example of data journalism by ProPublica was Mes-sage Machine where they first, collected data from their usersabout the recent scandal of mails being sent by the Obama Cam-paign to Twitter users. ProPublica collected a small dataset ofthese mails and showed in a simple yet effective way how mailssent out to different mails were related. This shows an exampleof how data can be effectively collected analysed and then de-livered to the audience such that it brings to the fore somethingmore than initially met the eye.

13. Do No Harm, Las Vegas Sun[12]: Las Vegas Sun is a daily news-paper in Las Vegas, Nevada, USA and won the Pulitzer Prizefor Public Service in 2009. They ran a campaign called “Do NoHarm” in 2010 on hospital care where they analysed hospitalbilling records in the order of millions. The results showed thatthere were over 300 cases where the patient died as a resultof surgical mistakes. The infographic used to depict their re-sults had various elements such as interactive graphics to see inwhich hospitals the cases were most prevalant, maps to showthe spread of infection from hospital to the other. The intituitiveand easy to navigate display of data not only made it comfort-able for the users to access the same, but it also helped show realimpact in the case with the Nevada Legislature taking action.

2L A N D S C A P I N G S T U D Y

A field trip was undertaken in late August 2012 to Ranchi, Jharkand.This was done in order to get an on-ground idea of the working ofJMV and also to understand closely the mobile phone skills of thecommunity reporters. Also in this field trip, discussions were heldwith the various partnering NGOs of JMV and also other stakehold-ers. The initial ideas regarding aggregating stories for better insightand those related to data journalism were discussed with the differentstakeholders. The discussion with the community reps to understandtheir mobile handling skills was necessary in order to design the appin such a way that it would be easy and comfortable for them tooperate it and still provide the necessary features. In all of these dis-cussions, a lot of valuable feedback was also obtained on the currentstate of the Goonj project along with ideas and suggestions on howto improve it.

Discussions were held with existing Mobile Vaani community part-ners in Jharkand such as Gram Swaraj Abhiyan and Public Health Re-source Network (PHRN). Both the organisations recognized the needfor better issue tracking and ensuring that problems reported don’tget lost in a big pool of incoming items. It was considered especiallyimportant from the point of view of follow-up on the grievances re-ported on a community-based system like JMV. Having themselvesworked in the social sector they had some invaluable suggestions.People at PHRN stressed that in order to make sure that an effec-tive follow-up is carried out, it is necessary to have a limited num-ber of tasks at hand at a given point of time. This needed that therelated items reported at JMV be clubbed together and seen contextu-ally rather than simply individual unrelated items.

As mentioned in the previous chapter, the items coming in at JMVare not only associated with an issue but also with a broad topic towhich it belongs such as agriculture, health, crime and others. Ideasabout these organisations getting the stories of a particular topic, theone on which they focus, via mailing lists were also discussed andnicely considered. Social workers at Gram Swaraj Abhiyan even sug-gested providing content from their side, such as messages issued inpublic interest which they would want to convey to the people viathe JMV number.

We also visited the office of Prabhat Khabhar in Ranchi. PrabhatKhabhar is a daily newspaper in Hindi and circulated in Eastern In-dia. We discussed the need for moderation of the incoming items atJMV and the issues of scalability of the same. The editor there stressed

9

10 landscaping study

on the need for moderation in citizen journalism platforms and alsothat its equally important that the items be published as soon as pos-sible since they come in. Discussion also lead to the point of hiringmore moderators coming up to do the same but it needing signif-icant time and effort in training them so that they can responsiblyhandle all the activities needed for the task of moderation. Certaindedicated community reporters which go and meet various peopleand help them record audios ensures that those items are recorded sothat they are ready to be published. Though still the need to scale interms of moderation requirements with increasing incoming calls isdefinitely a challenge and one that needs a solution. We showed himthe Goonj application where the community representatives recordaudios, videos and photos. He appreciated it for the additional formsof media it could help contribute to JMV.

Last but not the least, we also met a locally active social workerdealing with grassroot level problems in Ranchi and discussed withher the concept of the JMV and what we were trying to attain withthe same. She listened to us patiently and appreciated the idea butonly had one point to make, which was that the usefulness or benefitof a system designed to help people is determined by the impact itshows and thats where we should focus.

We also met with six community representatives of JMV. Three ofthem had been allotted the Android phones with the Goonj app andhad been using the same to collect stories for JMV. The other threewere using basic feature phones and used those to help people recordmessages by calling in on the toll free number. The discussions withthese people were particularly important in understanding the associ-ated emotions of the actual people who use the JMV platform to bringto the fore what they wish to convey. We were informed that peoplewait eagerly to listen to an item they have recorded and have a feelingof immense pride when that item does appear. On the flip side, theyget disappointed if an item recorded by them did not get published.This lays emphasis on the need to give guidance calls correspondingto every item that the moderators did not find fit to publish in orderto inform the caller the reason why it was not published and whathe/she should take care of from next time.

Other than this, after having recorded a grievance with JMV peo-ple expect some action to be taken on the same. This also hints to-wards the need of better issue tracking, the same concerns whichwere raised by all the other stakeholders. Therefore there is a definiteneed to be able to make sure that the stories reported have a properfollow-up later and we are able to recognize the main issues out ofthe items being reported. Also there is a need to present all of this in-formation in a way so that the authorities can be shown all the detailsin a clear and concise manner and some true impact can be delivered,as stressed on by the grassroot social worker we met.

landscaping study 11

In order to get an idea of the mobile handling skills of the commu-nity reporters, we asked them to perform certain tasks on the Androidphones they were using. These included the following:

1. Making a call

2. Sending an SMS

3. Navigating to an app not on the very first screen on the phone

4. Changing settings of the Facebook app

5. Accessing folders on the Android phone

Also in the days spent in Ranchi we observed their smartphoneapp handling capabilty as well. The main points observed are be sum-marised as follows:

• All of them could make calls from their mobile phones, andsend SMSs too but everyone said they prefer making a call morethan sending an SMS as it seemed significantly easier to them.

• All of them could handle 4-5 levels of hierarchy in menu brows-ing.

• As a result all of them could navigate to any desired app onthe phone and even to where the folders are located (once toldwhere).

• All except one did not understand there were more optionsthat would appear on pressing the left hardware button on thephone.

• Soft keys were not clearly understood by them. That is all exceptone could not understand that the left hardware button whenpressed would give different options at different times.

• Most of them were comfortable with English but suggestedkeeping the interface for the app in Hindi only.

• All except one were used to using internet on their phones.

3T H E H I E R A R C H I C A L M O D E R AT I O N M O D E

As stated before, in a community sourced information medium, theneed for moderation before the content can be published is indis-pensable. As call numbers rise, there is an increased burden on themoderators at central level as there is a need to publish the incomingitems as soon as possible. One approach is to hire more moderatorsand another is an heirarchical approach which this project tries toexplore.

3.1 moderation data analysis

In order to make a better decision of how to split the responsibilitiesof moderation across the two levels, a rigorous exercise was carriedout by the currently working dedicated team of moderators for aperiod of five days. In this exercise, they recorded how much time wasspent in the various moderation activities. The different activititiesthey considered are as follows:

• Listening to the recorded item.

• Giving appropriate tags. This is in terms of location, topic/cat-egory and also whether it is a junk call, whether it needs audioediting, needs a guidance call among others. Guidance calls aregiven to the contributors of the stories in case the call needsto be recorded again. This can be due to various reasons (po-litically motivated item, incomplete message, quality or contentissues, etc.).

• Giving the guidance calls, if required.

• Audio editing of the items, if required.

• Writing transcripts for the audio recordings, which are usefulfor the people accessing the news item through the web inter-face.

Therefore, an item being reported can either be a junk call, need aguidance call so that its re-recorded, need some audio editing or itmight be directly publishable. For all items that are being published,appropriate topic/location tags have to be given and the transcriptswritten. The results obtained after analysis of the data are in the nextsection.

13

14 the hierarchical moderation mode

Category Number of Items Percentage

Junk Calls 92 24.97

Audio Editing Needed 26 7.01

Guidance Call Given 58 15.63

Unpublished Items 123 33.15

Published Items 240 64.69

Total Items 371 100

Table 1: Proportion of items in different categories.

3.1.1 Analysis and Results

The proportion of items belonging to different categories are sum-marised in Table 1. The results of this exercise are summarised asbelow:

• About two-third of the items coming in are actually published,rest are unpublished as they are either junk calls or need to bere-recorded due to various reasons.

• Close to a quarter of the calls are junk calls, while those requir-ing guidance calls are only 15 per cent of the total calls andthose needing audio editing are only 7 per cent of all the incom-ing calls.

• The average time spent by a moderator on giving a guidancecall is about 2 mins per call. For those items which need a guid-ance call, this takes about 32.32 percent (nearly one-third) of thecall spent in moderating the items.

• For items marked as needing audio editing, the activity of edit-ing took, on an average, 31 percent of the total time to mderatethose items. This is also close to one-third of the time spent inmoderation.

• Tagging of items to be published too close 20 percent of the timeto moderate those items.

Based on the results from this exercise and insight from the fieldtrip to Ranchi, a model for heirarchical moderation was designedwhich is explained in the next section.

3.2 moderation through the goonj app

In this section, we explain in detail the heirarchical moderation pro-cess and the app designed to with the help of which the burden of

3.2 moderation through the goonj app 15

moderation is shared by the central team and the community repre-sentatives responsible for data collection. But before a delve into thedetails, we give a small explanation of the Goonj app, to which thevarious features of moderation are added.

3.2.1 The Goonj Mobile App

In the current scheme of JMR, the data collection process is two-fold.Not only are stories contributed by citizens calling in at the JMR num-ber, but also recorded via the Goonj Mobile App used by few com-munity representatives on an Android phone. The app enables themto record audio, video and also click photographs, hence broadeningthe types of media available at JMR. Moreover these community rep-resentatives also take care of the follow-up processes by personallycommunicating with the responsible authorities regarding differentstories reported at JMR. This approach ensures that the problems sur-facing through JMR also see some appropriate action being taken.

3.2.2 Distribution of Moderation Responsibilties

While it is important that the responsibilties of moderation be splitacross the two levels so that there is maximum reduction in load onthe central moderators i.e. those activities which serve as the mosttime consuming should be designated to the community representa-tives, it is also important to design taking into account their limita-tions in terms of their mobile phone handling skills. Keeping boththese factors in mind, we decided to follow the distribution as shownin Figure 1.

As can be seen, we kept the responsibility of making guidance calls,tagging and marking what category the call belongs to at the firstlevel (of the community representatives). Due to such a division, allthat is needed by them to do on the mobile is to select one of manyoptions visible after listening to the audio of the call, and making aguidance call if required. As discussed previously in Chapter 2, bothof these are functionalities that they can handle with ease. Also thejunk calls and those needing guidance together comprise nearly 40

percent of all the items which need moderation. This helps shift aconsiderable amount of work to on the level of the community rep-resentatives without demanding much from them in terms of theirskills and also simplifies the job of training them to use the app. Onthe other hand, the task of audio editing was maintained at the cen-tral level itself, as first those items comprise only 7 percent of allitems, and that would need considerable training for the communityrepresentatives and also need more sophisticated tools to be madeavailable at their mobile phones.


Figure 1: Distribution of responsibilities across levels.

3.2.3 System Flow

The design provides a facility for crowd sourced moderation as anenhancement over the existing Goonj app. A flow diagram of thesystem design is shown in Figure 2. The working of the entire systemis as explained below :

• The calls coming in through the JMR server are pulled into amoderation database. Every call is then delegated to one of thecommunity representatives with the help of an admin interfacedesigned specifically for the same.

• The items delegated to a community representative are thenpulled to his mobile and are accessed by him/her at a suitabletime.

• Once the community representative has moderated the item, itis then sent back to the moderation database.

• The central level moderators can then see the tags given at thefirst level with the help of the admin interface and complete therest of the moderation process like transcripting etc.

The syncing of stories assigned to a community representative fromthe moderation database maintained on a server to his mobile phoneand back is done using the same flaky internet framework throughwhich the app sends the audios/videos/photos recorded by the com-munity representative. This ensures that the stories assigned to a com-munity representative are definitely received at his phone even if he

3.2 moderation through the goonj app 17

Figure 2: Flow of data in the system.

Figure 3: Admin interface:Page showing list of stories and their status


is in an area where internet connectivity is poor. Also this helps en-sure that the effort put by him in moderating an item is not lost whenbeing sent back to the server due to poor internet connection.

3.2.4 The Moderation App

The app has been developed and tested on Samsung Galaxy S2 andalso on comparitively low-price android phones such Samsung Aceand Samsung Fit. Figure 3 shows some of the screenshots of the An-droid app designed. The original components of the app throughwhich the community representatives used to contribute the audios,photos and videos has been kept unchanged so as not to confuse thecommunity representatives and make it easier for them to understandthat the moderation feature is just an "addition" to the app they wereusing till now, and rest is the same. A sense of familiarity with theapp helps them feel more confident about using the app. The app hasbeen maintained to be completely in Hindi.

Figure 4: Screenshots of the Goonj app

The steps to be followed for the community representatives to mod-erate through the app are as follows:

• The very first screen of the app shows an option of "Moder-ate Items". Hence whenever the community representative findssuitable, he can click on that tab which leads him to a screenwhich saya "Click to fetch an item to moderate".

• On clicking that button, he/she is led to the second screenshown in the screenshots in Figure 4. There he can play theaudio of the item by clicking a button and listen to the same.

3.3 tracking performance of community representatives 19

He can then select one of the four options given below whichleads him to different screens:

– Junk Call : If the community representative clicks on thisoption, the moderation process for this item at the firstlevel is complete and hence we go back to the screen wherehe can fetch a new item to moderate.

– Guidance Call Needed: On selecting this, the communityrepresentative is taken to the third screen in Figure 4. Herehe can make a call to the contributor of the item by simplyclicking the "Call" button. The time for this call is recordedby the app. After he finishes the call, he is asked whetherthe guidance call was successfully made. To this he cananswer by clicking either "Yes" or "No". On clicking yes,the moderation process for this items ends. If he clicks no,then the item is stored so that the call can be made againlater

– Audio Editing Needed: If this option is clicked then thisitem will be published after audio editing is done at thecentral level. So on clicking this, the community represen-tative is taken to a screen where he is asked to give locationand topic tags to the item. This is done by selecting one ofthe topics or locations shown to him as a list. Once he hasdone that the moderation process ends.

– Publishable: In this case as well, the community represen-tative is taken to the screen which allows him to assigntopic and location tags by selecting one from a list. Oncethis is done he is directed back to the main screeen.

• The first screen on the app also shows a button saying "GiveGuidance Call". Selecting this leads to a list of all those itemswhich were marked as needing guidance call by the communityrepresentative but the call did not go through or was not madesuccessfully at the first time the item was encountered. Selectingone of these story items, takes the community representative tothe third screen in Figure 4 and the process from there is sameas explained above.

3.3 tracking performance of community representatives

Such a moderation scheme definitely reduces the burden on the cen-tral team, but there are also certain risks involved. For instance, thecommunity representatives might not moderate the items delegatedto him on a given day at all. Also the accuracy of their moderateditems is another aspect. They might irresponsibly mark a publshableitem as junk or mark an item not worthy of publication as publish-able. In another situation it might happen that there is a story which


has an urgency to be reported and for some reason, the concernedcommunity representative is not able to moderate it, the central teammust then have the facility to assign this item to another communityrepresentative. Also there should be a quick way to see which com-muntiy representatives are most responsible so that they can respondto the urgent need.

Therefore, in order to ensure that the community representativesare doing their job well, we maintain a dashboard for each one ofthem in order to track their performance. The same admin inter-face which is used to delegate items to community representativesis given an extra tab as shown. This lists all the community repre-sentatives and some statistics regarding their performance alongwiththeir names in separate columns. These statistics are:

1. Number of items moderated given the number assigned

2. Number of guidance calls made given the number of itemsneeding guidance calls

3. Number of items marked publishable by him and found so bythe central moderators as well

4. Number of items (audio, video, photo) contributed via the Goonjapp

4R E A L I T Y R E P O RT I N G

An issue or a problem once observed through JMV helps spreadawareness and search for solutions. The concerned authorities needto be contacted and action needs to be taken. Thus to make the re-porting of a story at JMV effective, constant follow up on that is alsoneeded. With the current simple design of the Goonj app, it is diffi-cult to keep track of the status of a particular issue once reported andtrack whether a follow-up was taken or not. The stories listed in alinear, unorganized fashion tend to get lost (as told to us by one ofthe community representatives). Also with an increasing number ofstories being reported at JMV, it is important to be able to link up sto-ries which carry related information so that they can be viewed in acontextual manner for a more comprehensive knowledge expereince.

In order to facilitate the same, the following three steps were taken:

1. Creation of an admin interface at the central level to create is-sues from the JMV items:

This is done as another tab in the admin to delegate items tocommunity representatives, similar to the tab to track their per-formance. A snapshot of the same is shown in Fig 5 where thecentral moderators can create a new issue with the help of theAdd button. Then they can pick and relate stories from JMV tothis issue. They can also add a description to the same for bet-ter clarity and assign media files such as photos and videos thatmight give more insight about what the issue focusses on. Thedifferent tabs on the same admin interface help in keeping vari-ous functionalities needed by the central moderator at the sameplace, yet keeping them separated in different leaves makingtransition easier.

2. Addition to the Goonj app:

From the admin interface described above, stories related to anissue can be added by the central moderators. But we wouldalso want that as the community representatives moderate newitems, they should also be able to associate a story with an issueor an older story that might have been reported and is relevantto the one at hand. Therefore there is a need to add function-ality to the Goonj app to allow this. But this spawns a numberof new design questions. Can the community reps create anissue by themselves and then add the concerned story to it?How should the functionality be added to the app so as not tooverwhelm the community representatives? It is important to

21

22 reality reporting

Figure 5: Admin interface to manage and create new issues.

remember that the community representatives are more or lessnovice smartphone users and the app they are used to handlingon that is a very basic one with a single screen allowing themto either click a photograph, record an audio or a video.

Keeping all these in mind, the following layout was finalized.The community representatives do not have the authority tocreate new issues. This design decision helps take into accounta number of things. For one, this makes sure that the number ofissues don’t blow up as they will only be created at the centrallevel by highly trained moderators. Another aspect is that dueto this there is no need to ensure that the apps running on thephones of the various community representatives to be kept insync, that is if an issue is created at one of the phones, then itshould also be visible on the other phones. Last but not the least,this greatly helps simpilfy the app interface for the communityrepresentatives. The functionality to associate the story beingmoderated is as simple as clicking on another button on thesame screen from where they give location and topic tags. Thescreen for this is shown in Fig 6 . When they click on the button,a list of the issues created at the central level is fetched anddisplayed. He/she can then select one of them to associate thestory being moderated with the same.

3. Building appropriate web interfaces for better presentation ofthe issues:

As discussed in the introductory chapter, suitable aggregationof stories and visualisation of the same are increasingly gainingimportance. Different types of interfaces for different issues canenhance the expereince and help in better extracting the main

reality reporting 23

Figure 6: App Screen to assign topic, location, issue tags.

points of an issue which has featured on JMV. In this project, wealso aim to develop similar such interfaces for the issues created.The Data Jorunalism Handbook[3] explains where it might beuseful to visualise data: to track changes over time, to comparevalues, to show connections, to trace flows among other things.The stories collected at JMV give ample oppurtunities for thesame.

A timeline based interface is shown in Fig 7, which is useful forthe issues which can be traced out as related events over time.For instance, in order to depict the progress of a NREGA workover time: demand being raised by the community, enrolling ofpeoplel, start of the work, work completion, maintenance anduse over time. This kind of an interface is particularly usefulfor follow-up on grievances reported at JMV: the problem beingreported, the voice of the people reaching the concerned author-ities, action being taken and people getting relief. The timelineshown is interactive with a self adjusting scale based on thegaps between the events related to an issue. The selected issueis shown in the middle and the user can go through all the in-cluded stories in a serial manner using the left, right arrows orjump to a particular event of interest by directly clicking on it.

JMV runs a number of campaigns concerning issues relevant toits users. For example, a campaign to spread awareness aboutHIV-AIDS, regarding Water Conservation and similarly Rural-Urban Migration Campaign among many more. The stories re-lated to these can also comprise an “issue” as they all are relatedand talk about the same topic. An interesting interface for themcan be one with a map showing the participating districts asshown in Fig 8. Also such campaigns collect a number of storiesand there is an oppurtunity to extract trends and various results.

24 reality reporting

These help to serve exactly the purpose of giving the user moreinsight that what was simply reported in an event-based fash-ion. For instance, for the Rural Urban Migration campaign runat JMV, the reasons of migration, the different problems facedby people can be depicted using interactive pie charts.

Several interesting visualisations can be created from the datacollected at JMV. An example of the same is shown in Fig 9.The stories coming in are given appropraite topics tags by themoderators. We club various topics into eight broad categoriesas shown. The infographic in the figure shows by the mediumof size of the bubbles what are the most popular topics aboutwhich stories are reported at JMV. Such a visualisation makesit very simple to understand that Governance and Culture andEntertainment are the most common topics, closely followed byHealth, Education and Livelihood.

Figure 7: Interface for Issue Coverage with timeline.

reality reporting 25

Figure 8: Interface for Issue Coverage with map.

Figure 9: Visualisation of different topics covered at JMV

5VA L I D AT I O N O F T H E H I E R A R C H I C A L M O D E L

In order to determine the usability of the designed application andvalidate the hierarchical moderation model as described in detail inChapter 3, the app was tested with three community representatives.The app was installed on the Android phones of the same three com-munity representatives in Ranchi which were earleir using the basicGoonj app. A total of 74 stories were assigned to the three commu-nity representatives over a period of five days. The number of storiesassigned to the moderators were 2 each on the first day, 5 each for thenext 2 days and 6 each for the last two days. This gradual increasehelped them to easily adapt to the task at hand. These numbers aresummarised in table 2. The results in the following sections are de-rived from the moderation results from these 74 delegated stories.The process followed is summarised below:

• Every day in the morning, one of the dedicated moderators ofJMV assigned the given number of items to each of the commu-nity representatives with the help of the admin interface.

• The delegated items were synced to the phones of the commu-nity representatives. Through the day, they would fetch theseitems and moderate them.

• The results of moderation were synced back and displayed onthe admin interface again.

• Since JMV is a live project in action, and this an experimentalvalidation test, the delegated items were moderated as usual bythe central team of moderators at JMV. These results were thenlater collected and compared with the results obtained from themoderators in order to obtain the insights stated in the nextsection.

5.1 results and insights

In this section the data collected over five days of run of the hierarchi-cal moderation process is analysed keeping in mind various aspectsnecessary to validate the same.

5.1.1 Accuracy

In order to measure the accuracy with which the community repre-sentatives moderated the items assigned to them, information was

27

28 validation of the hierarchical model

Total number of delegated items 74

Number of community representatives 3

Number of days of test run 5

Number of delegated items (day1) 6

Number of delegated items(day 2 and 3) 32

Number of delegated items(day 4 and 5) 36

Table 2: Moderation app test run

collected first about the status tags(junk, needing guidance call, need-ing audio editing, publishable), location and topic tags given by themoderators at the central level. As per this, out of the 74 delegateditems, 33 were marked as ones needing guidance calls and the rest41 as being publishable. Discussions with the moderators revealedthat there were no items which were marked as needing audio edit-ing as they prefer now to mark those items as needing a guidancecall itself, so that people are also made aware of what all things theyneed to take care of so that the audio recording is clear, given theyhave a message to convey. Also there were no items marked as beingjunk as similarly blank items, all items with an incomplete message,or someone trying to use the system but being unsure how are alsomarked as being needing a guidance call as they shoudl be guidedbased on whatever problems they faced. Junk calls are only given tothose items where someone is intentionally trying to be abusive, orplaying a prank and none such items were considered fit t be dele-gated to the community representatives in the 5 days of the run ofthe process.

These moderation results were compared to those received fromthe app running on the phones of the community representatives. Theaim was to figure out in what fraction of the stories, the category as-signed by the community representatives matched the one assignedby the central moderators. Also for those items which were consid-ered publishable by both the level of moderators, how well did thelocation and topic tags match. The comparison led to the followingresults:

Accuracy in marking the status tag = 45.95 percent (34/74)Percentage of items needing guidance correctly identified = 54.54

percent (18/33)Percentage of items considered publishable correctly identified =

39.03 percent (16/41)Accuracy in marking location tags = 65.85 percent (27/41)Accuracy in marking topic tags = 51.22 percent (21/41)These numbers seem very low in terms of the accuracy required

for a model like this. In order to gain more insight, a matrix was

5.1 results and insights 29

Figure 10: Accuracy in correctly identifying different categories

Categories Junk Guidance Call Needed Audio Editing Needed Publishable

Guidance Call Needed 6 18 2 7

Publishable 6 3 16 16

Table 3: Confusion Matrix among the various categories

built based on the ideology of the confusion matrix, which is oftenused in supervised machine learning to figure out the classes amongwhich there is maximum confusion by the algorithm at hand. Thesame is shown in table 3. The rows of the matrix correspond to thecategories in which the items were classified by the central moder-ators. As explained above, these were only two. The columns of thematrix correspond to the categories in which the items were classifiedby the community representatives.

If we look at the first row in table 3, the total number of itemswhich needed a guidance call which were either marked as beingjunk or needing a guidance call is 24 which is 72.73 percent of thetotal items needing a guidance call. On listening to the 6 items in thiscategory which were marked as junk, it was found that they werethe ones with an incomplete message or blank items. Therefore thismeans that the community representatives were able to recognize thatthis items is not useful and hence marked them as junk, while theyshould be marked as guidance call needed so that instead of justdiscarding these items, we can help the callers at JMV to understandthings better and hence increase the caller database. Therefore, seenwith this perspective, the accuracy to recognize items which are not fit


for publishing is close to 72 percent but there needs to be more cleartraining to make the community representatives understand that inwhat all scenarios a guidance call is still needed and which items aretruly junk calls.

The second row in table 3 corresponds to the items considered pub-lishable by the central moderators. The accuracy to recognize itemsin this category by the community representatives is significantly low(39.03 percent). But a look at table 3 shows that bulk of these items(when not considered publishable) were marked as needing audioediting by the community representatives. That is they were able torecognize for an impressive 78.05 percent (16+16/41) of the publish-able items that these items have a clear message to depict and hencethey provided the requisite topic and location tags as well for theseitems. Such a high occurance of marking publishable items as need-ing audio editing seems to be the result of a conservative approachby the community representatives in marking items as publishable.Therefore even minor faults in the quality of the recorded audios,which as per the central moderators can be overlooked to publish theitems, are being recognized by the community representatives andmarked accordingly.

A surprising result though revealed from table 3, is that of 6 of thepublishable items, close to 15 percent of the total, were marked asjunk by the community representatives. This is surprising given theability demonstrated by the community representatives to be able todistinguish items with a clear message from those that did not carryone.

The accuracy with which the community representatives markedthe location from where the story was contributed is around 66 per-cent i.e. in two thirds of the cases they were able to recognize thecorrect location. The accuracy in the case of the topics though iscomparitively lower ( 51.22 percent). This can be attributed to thesubjective nature of assigning these topic tags as well. For example,an item reporting grievance regarding a health related governmentscheme can be considered as beloging to the “Health” category aswell and also the “Governance” category. Similarly, as seen from thedata, there were a number of clashes betweent the topic “Governance”and “Livelihood” category. As a result of such mismatches, the actualaccuracy number being reported is low. Mostly though the error inmarking topic and location tags is where the community representa-tives could not recognize the same from the audio and hence markedit as None.

5.1.1.1 Performance Specific to Community Representatives

As mentioned before, the app was tested with three community repre-sentatives. In order to be gain better insight about the overall low ac-curacy of marking the items delegated, an analysis was carried out to




Publishable 2 1 6 3

Table 4: Confusion Matrix among for crepB



Publishable 4 0 9 2

Table 5: Confusion Matrix for crepC

tap the performance in terms of accuracy of all the three communityrepresentatives individually as well. For the remainder of this section,the community representatives will be referred as crepA, crepB andcrepC. These three people differed in terms of their previous knowl-edge about the entire moderation process. crepA is currently part ofthe dedicated team of moderators for JMV. crepB on the other handhas been involved in the moderation process when JMV had startedoperation but since the past many months has not been involved withthe moderation process at JMV. crepC on the other hand was awareof the fact that the items reported undergo moderation but was intro-duced to the actual process of how moderation is carried out, whatare the facts to be kept in mind at the time the app was introduced tohim.

The overall accuracy in marking the correct status tag(junk call,guidance call needed, audio editing needed and publishable) are be-low:

crepA : 69.23 percentcrepB : 37.5 percentcrepC : 29.17 percentThese numbers indicate a stark difference in the accuracies ob-

tained by crepA and those obtained by crepB and crepC. While crepAhas an accuracy close to 70 percent, those of the other two do not evenreach 50 percent. Also we see the accuracy falls as the information thecommunity representative has of the moderation process decreases.As the accuracy of crepB and crepC is below that of chance, an anal-ysis similar to above was further performed on the items moderatedby them to figure out the cause of such low accuracy values. The con-fusion matrix tables for crepB and crepC are shown in Table 4 and 5

respectively.


Before delving deep into what Table 4 and Table 5 reveal, its bestto put down the accuracy obtained by crepB and crepC in the twocategories i.e. guidance call needed and publishable items.

Percentage of items needing guidance correctly identified (crepB)= 50 percent

Percentage of items considered publishable correctly identified (crepB)= 25 percent

Percentage of items needing guidance correctly identified (crepC)= 55.55 percent

Percentage of items considered publishable correctly identified (crepC)= 13.33 percent

Lets consider crepB first. The accuracies obtained by him are betterthan those by crepC but still are considerably low. For items whichneeded guidance calls, his accuracy in correctly identifying them is50 percent. Table 4 shows that most of the wrongly identified itemsin this category by crepB were marked as publishable, and only oneitem was marked as junk. Since items marked as junk and needingguidance call are the ones which are not useful, crepB could identifythe not-useful items with an accuracy of 58.33 percent (1+6/12) . Forthe items which were publishable, the accuracy for him in correctlyidentifying them is very low, just 25 percent. But table 4 shows thatmost of the wrongly marked items in this category were marked asaudio editing needed by crepB. He was able to realise that 9 (6+3)items out of the 12 items had a clear useful message and hence alsoprovided topic/location/issue tags for the same. This means an accu-racy of 75 percent in identifying useful items. This shows that thereis a need for better sensitisation of crepB to what audio quality is stillacceptable at JMV and which not.

Similar results are obtained for crepC. For items needing guidancecall, he still managed a fair accuracy of 55.55 percent, which is muchbetter compared to his overall accuracy and the ones in identifyingpublishable items which is at just 13.33 percent. But Table 5 showsthat 9 of the 15 publishable items delegated to him were markedas needing audio editing. Hence if we calculate his accuracy in beingable to identify a useful item and mark it correctly using the app thenit comes to an amazing 73.33 percent (9+2/15). Similarly accuracyfor the items needing guidance call is low as most of the wronglymarked items were treated as junk. The accuracy to in being able toidentify that the item cant be published comes at am impressive 88.88

percent (3+5/9). Hence if we see only the overall accuracy obtainedby crepB and crepC, it seems that crepB did a better job. But theconfusion matrices for them reveal that the ability to discriminatebetween useful and non-useful items is higher for crepC then crepB.These values are also depicted graphically in figure 11 .


Figure 11: Accuracy for crepB and crepC in different categories

5.1.2 Ease of use of the app

Discussions were held with community representatives regarding theease of use of the application, both in person when they had used theapp for a day or two and then later over phone calls. As per thesediscussions, all three community representatives found the app intu-itive. The fact that each one of the 74 delegated items were moderatedholds as a testimony to the same as well. Listed below are some ofthe comments of the community representatives after using the appli-cation.

One of the best things I like about the application is that there is noneed to manually pick up the number from which the item was

contributed. Once you listen to the audio, and mark it as needing aguidance call, all that we need to do is press the call button and talk

to the concerned person. Makes things very easy.Many a times it happened when scrolling the list of topics, issues or

districts that one of them got selected. Or when trying to selectRanchi, accidently the finger touched Bokaro which was directly

above. So this was a very good thing that only the last item selectedin those lists were updated with the story. This releived a lot of

stress in terms of handling the app.Mostly whenever we tried to call the concerned number to give

guidance, the call did not go through. One reason is that thenetwork is not so good here so many a times the lines are foundbusy. So the give guidance call button on the main screen is a big


help. If in poor network, we can locally moderate all the itemswithout waiting for the call to be necessarily completed.

One problem reported with the design of the application was thatwhen after listening to the audio, the community representative isasked to select either one of junk call, guidance call needed, audioediting needed or publishable, selecting the junk call ends the processof moderation there and then. Since an item marked as junk doesnot need any further processing from the community representative,from the point of view of the app there is no new screen to go to andthus the item is updated and we go back to the screen to fetch newitems. The other three options open a new screen and hence if theyget selected as a mistake, the community representative could pressthe back button and select the right option again. But in the case ofjunk call, once selected there is no way to undo that selection.

This might as well be one of the reasons why there were about 15

percent publishable items marked as junk by the community repre-sentatives.

5.1.3 Distribution of load

Since the hierarchical moderation process was run in an experimentalsetup, the items were moderated as before by the central team ofdedicated moderators. Therefore it is not possible to give results interms of how much exactly the model helped reduce the load on thecentral team. But we can surely project some numbers regarding thesame.

Referring back to the analysis of the time taken and composition ofthe items coming in at JMV as discussed in Section 3.1, we see thaton a daily basis, about 25 percent of the items are junk while about15 percent need guidance calls. Lets assume that a central moderatormoderates 100 items in a day. Due to the division of responsibili-ties, the junk calls are discarded by the community representativesand guidance calls made by them as well. This means that 40 (25+15)items have been handled at the first level of the hierarchy itself. Simi-larly for the remaining 60 items the tagging would have already beendone. As stated before, this takes about 20 percent of the time tomoderate an item. So the time to moderate these items is reduced byone-fifth of the total.

Average time to moderate a single item 7 minsTime taken to moderate 100 items = 700 minsTime taken to moderate 60 items as usual = 420 minsTime taken when tagging is handled by the community representa-

tives = (0.8)*420 = 336 minsTherefore the central moderator who was eariler achieving the sta-

tus of 100 moderated items in 700 mins, can now do so in 336 mins,which is 48 percent of the time taken before. Therefore we observe

5.2 observation, challenges and suggestions 35

that the theoritical benefit that can be attained by this model is a re-duction of more than half the load on the central team. But clearlythere are various issues which reduce this benefit as have been dis-cussed before. In the next section, we also see some more challengesthat this model might face.

5.2 observation, challenges and suggestions

In this section, we summarize some of the observations made in thelast section, cite some challenges observed at the time of putting theapp in action and suggest some ways to counter the same.

5.2.1 Observations and Conclusions

The major observation from the previous section are summarized be-low:

• The analysis represented in table 5.2 revealed confusion amongthe community representatives in when to mark the item asjunk and when to mark the item as needing a guidance call.This calls for a need to train them to better understand the dif-ference between the two categories and make them understandthe importance of giving a guidance call whenever there seemsto be a genuine attempt to record a useful message but for somereason there is inability in doing the same.

• Similar analysis also showed confusion between the audio edit-ing needed and publishable categories. There is a need to sensi-tize the community representatives that what quality of audiocan be allowed as publishable and which not. Even then there isstill a chance of error here given the subjectivity of the problem.

• Even though the overall accuracy of marking the items is low,the ability of the community representatives to be able to distin-guish useful items from not useful ones is demonstrated.

• There were situations where publishable items were marked asjunk by the community representatives. One of the possible rea-sons for the same can be selecting the junk call button by mis-take. In order to deal with this, the app needs to be updatedwith a pop-up asking the community representative whetherhe is sure that he wants to mark the item as junk. Only when heselects Yes, the update should be made. This will further greatlyhelp reduce errors of this kind.


5.2.2 Challenges and Suggestions

The following are some of the challenges identified in the smooth runof the hierarchical moderation process. The ones listed below are pri-marily from the point of view of the app where inspite of rigoroustraining and best interest and efforts of the community representa-tives, their contribution is not visible at the central level to help aidthe moderators, specially at times where there is need to moderatesome items urgently so that they can be published as soon as pos-sibe.

1. Poor internet connectivity:

As has been stated before, the internet connectivity in the ar-eas where the app is operated is often not good. Even thoughmany a times there is network available on the phone, internetconnectivity speeds can be very low. The app has been built touse the flaky internet framework for the same reason, so thatthe updates can be sent to the central level in a delay tolerantfashion. But due to no internet connectivity or slower speeds,there might be delay in the time when the updates are receivedat the central level. In the five days that the app was in action,updates for items moderated by the community representativessometime in the afternoon are received at the server here inDelhi late at night.

2. Long Power Cuts:

A number of times, there are long power cuts due to whichthe Android phones cannot be easily charged again. Imaginea scenario where a community representative moderated itemsdelegated to him, but due to inability to charge the phone hisphone got switched off before the updates can be sent. In sucha case, updates won’t be received at the server till the phoneis switched on again and gets proper internet connectivity. Inorder to deal with this, the community representatives weretaught about simple steps to manage their battery. For example,keeping the phone brightness low, switching of apps which con-nect to the internet and regularly charging their phones when-ever they have electricity. These are small precautions that canbe taken to avoid a situation like above but it can still be a causeof delay in the updates being visible to the second level of mod-erators.

3. Novice Users:

Since the community representatives are novice smartphone users,problems such as Message Memory Full or Phone Memory Fullperplex them and they are not able to handle them. The way thephone functions is that you are not able to access anything on

5.2 observation, challenges and suggestions 37

the phone until some memory is freed. In order to counter suchsituations, basic guidelines were given to the community repre-sentatives in written on how to handle such situations. But thiscan only be done for a few specific common problems. This isnecessary as when these community representatives take theirphones to a local shop to be checked, at times they delete cer-tain applications. One of the community representatives we hadmet had faced this situation with the basic Goonj app and hencewas unable to contribute any stories since a long time, inspiteof best of intentions.

6W E B S I T E

The website for better and organised display of the stories is currentlyavailable at www.goonj.net . Figure 12, 13 and 14 show few snapshotsof the same.

6.1 technical specifications

The technical specifications of the website are summarized below:

1. The website is built on the Django platform, a high-level Pythonweb framework. The underlying database in use is MySQL.

2. Uses HTML5 and CSS3

3. Deployed on the Apache Web Server

Figure 12: Main Page of the website.

39

40 website

Figure 13: Webpage for the topic channel - Education.

6.2 details of the web pages

Home page:.The main page of the website shows the most recentthirteen stories reported at JMV with the latest one being the "topstory". It features stories both in Hindi and English, whatever is foundbest suitable by the admin for the title and transcript. Every story hasassociated topic and location tags. Also the admin (via the admininterface of the website), can create new issues and then delegate astory as part of an issue.

Reports page: This page shows some statistics related to JMV andalso various reports from GramVaani. There is a map depicting thenumber of calls coming in from various places across Jharkand andanother graph showing number of callers and incoming calls perweek. The reports on the page include various activity reports andalso ones for various social campaigns such as Rural-Urban Migra-tion Campaign, Water Conservation Campaign among others. It alsofeatures videos of talks and interviews by Gram Vaani’s CEO Aa-diteshwar Seth.

On the main website, therefore, stories are displayed organized interms of the following:

1. Topic Channels : Stories in Goonj are given one or more tagsfrom a set of 172 tags. Each one of these tags is mapped intocertain broad categories which are termed “Topic Channels".

6.2 details of the web pages 41

Figure 14: Webpage showing Reports and various statistics.

42 website

There are currently 8 topic channels: Culture and Entertainment,Crime, Agriculture, Health, Livelihood, Governance, Environ-ment and Education. These appear as a drop down on the mainpage and clicking on any one them, we are directed to the pageof that particular topic channel, which gives a short descriptionof the topic and lists stories falling under it in the form of a list,with recent items appearing first.

2. Location : This is to enable a user to see the stories which arecontributed from a particular location in Jharkand. Currently“Locations" on the main page gives a drop down menu of cer-tain locations, clicking on any one of which takes us to the pageof that location listing stories reported from that location in theform of a list.

3. Issues: The Issues tab on the main page redirects to a page list-ing all the issues, each one of which is a link to that particularissue’s page. Page for a given issue, gives the description of theissue, along with audio, video, and photos specific to the issueand a list of stories falling under the issue.

6.3 populating the database

There are 2 phases for populating the tables in goonjDB (the MySQLdatabase underlying the website).

1. One-time phase using MySQL dump : A MySQL dump of thedatabase used by http://goonj.net was taken and all of thestories that have been published or archived were added togoonjDB. There were overall differences in old schema and newGoonj website schema. In the previous database, 170 tags areused to tag stories with key areas. Looking at these tags givento each story, it was assigned a Topic Channel through a spe-cific mapping. Similarly there is a new table for locations inthe new schema. This separately maintains location informa-tion. So now stories were connected with a location record inthe location table. This mapping was made with regard to thelocation contained in previous database. The location assignedto each of the stories was first mapped to one of the locationsin goonjDB using a spell checking script. Since many differentspellings or representations exist for a particular location in rawdata, a spelling mapper classifies them into specifically namedlocations. Each of these locations was then connected to corre-sponding stories. Third and most important modification fromold database was connecting media with stories. goonjDB main-tains value of file location as an entry in the media columns.This location of a file on server was derived from certain pa-rameters within the table. With a query to retrieve data from

6.3 populating the database 43

desired location we populated the media into the database andcould therefore get audio with the story.

2. Automatically with help of script : As new stories keep comingin, they are stored on a remote server, and need to be pulled intogoonjDB. This is done with the help of a python script whichhas the following stepwise functionality :

• Send a GET request to a designated URL coupled with thedate and time this script was last run.

• In response, receive a JSON file consisting of all storieswhich have come in since the date and time passed as datain the GET request.

• If there are new stories which have come, rsync with theremote server to get a local copy of the audio files for eachof the stories.

• Parse the JSON data and insert values into the goonjDB.

• Add the corresponding story to topic channel mapping ingoonjDB.

• Also add the story to location mapping in goonjDB.

The same is depicted pictorially in Figure 15. This script is setas a cron-job which runs every half hour to ensure that the webinterface for Jharkand Mobile Vaani remains upto-date with thestories reported by people by calling in at the designated num-ber.

Figure 15: Step-Wise flow for the Python Script.

B I B L I O G R A P H Y

[1] S. Cohen, J.T. Hamilton, and F. Turner. Computational journal-ism. Commun. ACM 54, 10 (October 2011), 66-71.

[2] L. Findlater, R. Balakrishnan, and K. Toyama. Comparing semilit-erate and illiterate users’ ability to transition from audio+text totext-only interaction. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems (CHI ’09). ACM, New York,NY, USA, 1751-1760.

[3] J. Gray, L. Chambers, L. Bounegru. The Data Journalism Handbook:How Journalists Can Use Data to Improve the News. O’Reilly Media,July, 2012.

[4] J.D. Lasica. What is Participatory Journalism? Online JournalismReview, August 7, 2003.

[5] I. Medhi, S. Patnaik, E. Brunskill, S.N.N. Gautama, W. Thies, andK. Toyama. Designing Mobile Interfaces for Novice and Low-Literacy Users. ACM Trans. Comput.-Hum. Interact. 18, 1, Article2 (May 2011), 28 pages.

[6] P. Mudliar, J. Donner, A. Jain, and W. Thies. Emergent practicesaround CGNet Swara, voice forum for citizen journalism in ru-ral India. In Proceedings of the Fifth International Conference on In-formation and Communication Technologies and Development, pages159–168. ACM, 2012.

[7] A. Vashistha, and W. Thies. IVR Junction: Building Scalableand Distributed Voice Forums in the Developing World. SixthUSENIX/ACM Workshop on Networked Systems for Developing Re-gions, June 2012.

[8] http://www.merinews.com/

[9] http://www.guardian.co.uk/

[10] http://usatoday30.usatoday.com/news/snapshot.htm

[11] http://www.propublica.org/special/message-machine-you-probably-dont-know-janet

[12] http://www.lasvegassun.com/hospital-care/

[13] http://livingstories.googlelabs.com/

[14] http://www.techmeme.com/

45

46 bibliography

[15] http://news.google.com

[16] http://www.breakingnews.com/

[17] http://flipboard.com/

reality reporting and moderation apps …cse.iitd...reality reporting and moderation apps for...

Documents