ALESSIOCIMARELLIDatascientistatDataninja

jenkin@dataninja.it|@jenkin27

dtnj.it/erice14

InternationalSchoolofScienceJournalismTheDigitalWorld(Erice,June10th,2014)

akajenkin

PASTMasterDegreeinPhysicsattheUniversityofRome"LaSapienza"

MasterinScienceCommunicationattheInternationalSchoolforAdvancedStudies(SISSA-ISAS)inTrieste

PressofficerattheEuropeanLaboratoryforNon-LinearSpectroscopy(LENS)inFlorence

PRESENTFreelancedatajournalist,webdeveloper,opendataactivist,citizenscientist,...

Datajournalism&datavisualizationmadeinItaly

Youknowverywellhowitworks...:)

Astopic

Storiesabouttheedgeofscientificresearchandhumanknowledge.

Keyroleinrelationshipbetweenscienceandsociety.

Sciencejournalistcanbeawatchdogagainstfalsescienceand scientificfrauds.

Asmethod

Itwouldbeevidentin ,becausetheworkflowissimilartopoliceinquiriesorscientificresearch.

Manyinformationsfromdifferentsources,accountabilityproblems,hypothesisandproofs,trialanderrorcycles,andsoon.

Notonlyastory,butalsoadiscoveryitself...

Awordinabuzzwordsera

whenhisinvestigationisultimatelybasedon(ordrivenby)digitaldata,heacquiressuchprefix.

Ifajournalistwanttotelltheworld,andtheworldisnowmadeofdigitalandquantitativeinformations,hehastoacquireskillsinmanagementandinterpretationofdata,orhewillmissanopportunity.

Teamworkandmultidisciplinary

Nosefornews,publicinterest,intuitionbasedoncontestknowledge

Analyticalmind,mathematicalandstatisticalskills,intuitionbasedonscienceofnumbers

Teamworkandmultidisciplinary

Problemsolving,hi-techknowledgeinhardwareandsoftware,nerd(orgeek,ifyouprefer)mood

Artisticsensibilityandintuition,knowledgeinUserExperiencetheoryandtechniques

Miners,dustmen,researchers,andstorytellers

Publicsearchenginesordeepweb?Official5-starsopendataorwebspidersandscreenscrapers?Monitorandkeyboard,smartphoneandtouch,orbootsandmud?

Datashouldbereadbymachinesandnotbyhumans!Datasetscouldhideerrors,inconsistencies,lies...orshowonlyapartofastory.

Miners,dustmen,researchers,andstorytellers

Normalizationsandcomparisons,filtering,grouping,aggregation,correlations,...

Howtorepresentnumbersandrelationsamongnumbers?Yes,witharabicnumerals,butpicturesareworthathousandwords...aslongasyoukeepinmindthattherearefactsbehindthenumbers,and

(copyrightofTheGuardian).

Inmethod

Yourunintoadatasetandfeelthepresenceofapossiblenews...OR

...youhaveaninterest,anidea,athesis,soyouarelookingfordata.

Havingquantitativedataaboutaphenomenonmeansthatsomewherethereisa youhavetounderstand,test,verify...andinterpret!

Datathemselvescansuggestnewwaysforyourinvestigationorevenfalsifysomehypothesisorassumptions.

Commonsense,intellectualhonesty,professionalethics

Somerandomexamples

NewScientistAppstornadoeswarmingworldexoplanetsplancksealevel

TheTelegraphmapofwindfarm

SortingalgorithmsMeteorites

EarthJournalismNetwork

by GlobalEditorsNetworkHealth

(NYT)(ProPublica)

(ProPublica)Environment

(ProPublica)(CenterforPublicIntegrity)

(LaStampa)(LaStampa)

Astronomy(NYT)

Energy(PlanbureauvoordeLeefomgeving)

AmericanWayofBirth,CostliestintheWorldInsidetheGovernment'sDrugDataWhichEmergencyRoomWillSeeYoutheFastest?

NewYorkfloodsBreathlessandBurdenedWhenItalyisshakingItaly,adelicateland

Kepler’sTallyofPlanets

Biomassa

Researchdata,scienceworld,citizenscience

Hardsciencesandsocialsciences

Ok,neitherLHCpetabytesareforjournalists,norstatisticaldatafromepidemiologicsurveys.

But ,or(open),whynot?

Ifyouarenotspecializedinaspecifictopicorifyoulacktheknowledgeabouttheframework,youcanasktoanexpertyoutrust.

Youcanalsousenumbersnotinaninvestigation,buttotellacomplexstoryusinginfographicsandinteractivevisualizations.

Bibliographies,socialnetworksofscientists,infrastructures

Scienceisahumanactivityandanindustry(almost)likeanyother.

HowaretheEuropeanfundsinvestedinscientificresearch?Wherearethecentersspecializedinthetreatmentofspecificdiseases?Whysomewellknownmonitoringtechnologiesarenotusedinsomecountries?

Sensor-basedjournalism

Cheapelectronicsandsensors+

openhardware+

freeinformationsharing=

datafromstakeholdersotherthanscientists

It'searly,butpromising:SwissMakeOpenDataCampsJapanGeigermapat-a-glanceCitizenScience&Sensors

Ifyouhavedata,it'sbetterifyouknowhowtodealwiththem.

Ifyouthinkyoumayfindsomedata,it'sbetterifyouusethem.

Ifsomeoneusedata,it'sbetterifyoucancheckhisclaims.

Playwithdataisfunny!

Welcometothejungle!

Someexamples

PublicadministrationInternationalorganizationsNGOsCivicactivistsPressofficesLeaksSocialnetworksJournalisticsourcesSinglejournalistsOurselves...

Datamadepublicandreusable

(USA)(UK)

(Italy)(Indonesia)

...

Data.govData.gov.ukOpenDataHubOpenIR

Rememberthebuzzwordera?

Datafrombigscienceexperiments(Atlas,HumanBrainProject,...)

Socialnetworks(Facebook,Twitter,butalsoeBay,Amazon,...)

Maybeit'snotforjournalists,butit'sahottopic...

GoogleEarthEngine

Formachine,notforhuman

Thekeywordis !

Awell-formedtablerepresentastructureddataset.Alistoffacebookcomments,articlesofanewspaper,arecordedspeecharenotstructureddata(andsoarenotmachine-readable).

Italldependsontheformat

IfwehaveGladstoneGanderasbestfriend:spreadsheet(xls,xlsx,ods,csv,tsv);not-so-commongoodformats(xml,sql,json,shp,kml,...).

Ifwearenotsolucky:tablesorlistsinwebpages(html);simpletablesinwell-donepdfs(pdf).

IfwehaveMurphyasworstenemy:scannedimages,evenifinapdfwrapper(png,jpg,pdf);digitaldatabehindcomplexsearchengines.

Andifwehavethebestdataever,butunderclosedlicense?

Well-formeddatasets

Numbersarenumbers,stringsarestringsandnotnumbers,datetimemustalwayshaveasingleformat(ie.yyyy/mm/dd),localizationisimportant,nogendervaluesinnames'columnorsimilarmixings,everyelementsshouldbenamedwithaUniqueIdentifier(ID).

Datatypescomputerunderstands:integers(withsign,zeroincluded),floatingnumbers(withsign),datetime,charactersandstring(casesensitive),nullvalue(thestrangecaseofavaluethatstates"I'mnotavalue").

Andsimplecomparisonsarestrictlyequalities,alsoinstrings!

Aggregation,average,normalization,relativedifference,distribution,...

Asinglerule:correlationdoesnotimplycausation!Spuriouscorrelations:Correlated:

http://www.tylervigen.com/http://www.correlated.org/

Ataglance

Withgreatpowercomesgreatresponsibility

Thebasicideaisquitesimple:youhavequantitiesexpressedinnumbersandgeometricobjectsdefinedbydimensions(ie.radiusinacircle),soyoujusthavetodecidehowconnectyourquantitiestovisualdimensions.

Thereareseveral(un)commonchartsandendlesscombinations:scatterplots,lines,bars,areas,pies,donuts,bubblecharts,treemaps,wordclouds,alluvionaldiagrams,dendrograms,networks,streamgraphs,gauges,chorddiagrams,motioncharts,parallelcoordinates,sankeydiagrams,maps,choropleth,...

On thereisanendlesslistofexamples!d3js.orggallery

Buildingasimpledatasetoralargeandcomplexdatabasefocusedonatopicofpublicinterestleadstoavaluableproduct:thedatabaseitself,intendedasacollectionof(linked)dataplusmetadata.

Canapublicfrontendtosuchdatabase,designedforcitizens,journalists,stakeholders,beconsideredajournalisticoutcome?Ifjournalismisapublicgood,itcanbeaservice,notonlyaproduct...

Scraping"Copy&Paste"combo

forChromebrowserGoogleSpreadsheetfunction

forsimplepdfsPython(orotherlanguages)scriptsandlibraries

CleaningFiltersand"Find&Replace"toolsinspreadsheets

AnalysisPivottablesandsimplechartsinspreadsheetsDedicatedsoftwares(ie.open-source or )

Viz, , , , ,

, , , , , ,...

DataMinerIMPORTXML()Tabula

OpenRefine

QtiPlot QGIS

Datawrapper RAW GoogleFusionTables Tableau CartoDBinfogr.am easel.ly Timelinejs Timemapper StoryMap d3js

TinaCasagrand," ",TheOpenNotebook(2014)PaulBradshaw," ",Leanpub(2014)JohnMair,RichardLanceKeeble," ",abramis(2014)PaulBradshaw," "ClaireMiller," ",Leanpub(2013)NathanYau," ",Wiley(2013)SimonRogers," ",Faber&Faber(2013)JonathanGray," ",O'Reilly(2012)NathanYau," ",Wiley(2011)

Datajournalismforsciencejournalists

ScrapingforJournalistsDataJournalism

DataJournalismHeistGettingStartedwithDataJournalismDataPoints

FactsareSacredTheDataJournalismHandbook

VisualizeThis

Alessio"jenkin"Cimarellijenkin@dataninja.it

@

Dataninja

Q&A

SWIM

jenkin27

www.dataninja.itschool.dataninja.it

dataninja.it/newsletter

school.dataninja.it/qa

sciencewritersinitaly.wordpress.com

Hacking+Marathon=Hackathon

ESPAD(Europeanstudentsanddrugs): http://www.espad.org/en/

RASFF(EUfoodsafety): http://ec.europa.eu/food/food/rapidalert/

http://ec.europa.eu/food/food/rapidalert/

TheRapidAlertSystemforFoodandFeed(RASFF)wasputinplacetoprovidefoodandfeedcontrolauthoritieswithaneffectivetooltoexchangeinformationaboutmeasurestakenrespondingtoseriousrisksdetectedinrelationtofoodorfeed.ThisexchangeofinformationhelpsMemberStatestoactmorerapidlyandinacoordinatedmannerinresponsetoahealththreatcausedbyfoodorfeed.

dtnj.it/rasff2013

http://www.espad.org/en/

Thisisthereportfromthefifthdata-collectionwaveoftheEuropeanSchoolSurveyProjectonAlcoholandOtherDrugs(ESPAD).Itisbasedondatafrommorethan100,000Europeanstudents.Overtheyearsabout500,000EuropeanstudentshaveansweredtheESPADquestionnaire.Atotalof36countriesandregionshavecontributeddatatothe2011ESPADDatabase.Drugslistincludescigarettes,alcohol,cannabis,otherillecitdrugs,tranquillantsandsedativeswithoutprescriptions.

dtnj.it/espad2011