BaltimoreDataDay2017BaltimoreNeighborhood
IndicatorsAlliance
FreeandOpenSourceGISTechnologyWorkshopJonathanDandois,Ph.D.
14July2017
Preparedfor:
©Fearless|page2
Background
WhatisGIS?
GeographicInformationSystems
• GISisthetechnologythathelpscombinedifferenttypesofdatatohelpexplore,analyze,andcommunicatespatialpatterns.
• WithGIS,weoftencombinespatialandnon-spatialdata(tables),andstackmultiplelayersofdatatorepresenttheworld
https://www.in.gov/gis/gis101.htm
©Fearless|page3
WhatcanyoudowithGIS?
AnalyzeandCommunicate
• Analyze->Answercomplexquestionsinvolvingspaceandtime:
• Howmanypeoplelivewithin1mileofaparkandhowlonghavetheylivedthere?
• Communicate->makemapstodeliveramessage
• Howdowechoosemapstylestoconveyourmessage:colors,linethickness,transparency,patterns,symbols?
https://www.in.gov/gis/gis101.htm
©Fearless|page4
TypesofGISData
SpatialData– VectorData
Polygons:Neighborhoods,States,Counties,CensusTracts
Points:Events,locations,stores
Lines:Roads,streams,routes
©Fearless|page5
TypesofGISData
Non-SpatialTabularData
DatatablesfromCSVs,Excel,etc.
©Fearless|page6
TypesofGISData
Non-SpatialTabularData
DatatablesfromCSVs,Excel,etc.
WhatinformationinadatatablecanweusetoRELATEtoourmapdata?
©Fearless|page7
AdditionalResourcesforintrotoGIS
• PennStateUniversityOpenCourses (http://open.ems.psu.edu/courseware)
• Basic:Geography160:MappingourChangingWorld(https://www.e-education.psu.edu/geog160/)
• Complex:GEOG585:OpenWebMapping(https://www.e-education.psu.edu/geog585/)
• QGIS(QuantumGIS)Tutorialshttp://www.qgistutorials.com/en/
©Fearless|page8
ToolsandSoftware
MainTools
GISSoftware
• QGIS(QuantumGIS)http://www.qgis.org/en/site/
• DesktopGISApplicationformapping,cartography,analysis
OfficeTools(somethingtomanagespreadsheetsandtables)
• LibreOffice (https://www.libreoffice.org/)
• FreeandopensourceOffice
©Fearless|page9
ToolsandSoftware
OtherTools
• OSGeo ‘Suiteofgeotools’http://www.osgeo.org/
• GRASSGIS– GISformorecomplexanalyses
• Geoserver /MapServer forsharinglotsofdataontheweb
• OpenStreetMap (mapwebsite,downloadbasemaps,useopengeotoolslikeageocoder)
• ESRIsuiteoftools– usewhatyouhave.
©Fearless|page10
EXAMPLE!CookingClass
Whereare311callsforrats?
Combine311tabledatawithcensustracts.Mapby:‘callcount’pertract.
©Fearless|page11
Getthingssetup
Installprograms
GetandinstallQGISandLibreOffice
QGIS(QuantumGIS)http://www.qgis.org/en/site/
LibreOffice https://www.libreoffice.org/
©Fearless|page12
GetData!
Whatformatsofdataarewelookingfor?
SpatialData
• Shapefile(SHP)- oldformatfromESRI,usuallycomesasazipcomposedofmanyfiles
• GeoJSON (GeoJavascriptObjectNotation)allthedataisstoredinasingletextfile
Others(TopoJSON,geodatabase…)
©Fearless|page13
GetData!
Whatformatsofdataarewelookingfor?
TableData
• CSV(commaseparatedvalues)
• Excelspreadsheets
• TXTtextfile
©Fearless|page14
GetData!
CensusTracts
StartatBaltimoreCityOpenData
1. https://data.baltimorecity.gov/
2. Searchforcensustracts
©Fearless|page15
GetData!
CensusTracts
©Fearless|page16
GetData!
Unzipthefile
Allofthesefiles(exceptforthezip)makeuptheshapefile!
©Fearless|page17
LoadthecensustractsintoQGIS
©Fearless|page18
LoadthecensustractsintoQGIS
©Fearless|page19
LoadthecensustractsintoQGIS
©Fearless|page20
LoadthecensustractsintoQGIS
©Fearless|page21
LoadthecensustractsintoQGIS
Onlyselectthe.shp file,clickOpen.
©Fearless|page22
LoadthecensustractsintoQGIS
©Fearless|page23
LoadthecensustractsintoQGIS
©Fearless|page24
LoadthecensustractsintoQGIS
©Fearless|page25
GetData!
311Data
https://data.baltimorecity.gov/City-Services/311-Customer-Service-Requests/9agw-sxsr/data
©Fearless|page26
GetData!
Startbyapplyingsomefilterstogetthedatawewant
©Fearless|page27
GetData!
©Fearless|page28
GetData! AddfilterforSRType contains‘rat’
©Fearless|page29
GetData! AddfilterforCreatedDate after7/7/2017
©Fearless|page30
GetData! Download/Exportthedata
©Fearless|page31
GetData! LoadCSVintoLibreOffice
©Fearless|page32
GetData! LoadCSVintoLibreOffice
©Fearless|page33
GetData! LoadCSVintoLibreOffice
©Fearless|page34
GetData! LoadCSVintoLibreOffice
©Fearless|page35
GetData! LoadCSVintoLibreOffice
©Fearless|page36
Wehaveneighborhoodinformation:
SoifwehaveamapofNeighborhoodnames,wecouldtrya‘TableJoin’http://www.qgistutorials.com/en/docs/performing_table_joins.html?highlight=table%20join
Wehaveaddresses,sowecould‘geocode’(assignmaplatitudeandlongitudecoordinatestoaddresses:https://www.census.gov/geo/maps-data/data/geocoder.htmlhttp://wiki.openstreetmap.org/wiki/Nominatim
GetData! HowdowemakethisdatareadyforGIS?
©Fearless|page37
Butwealsohavea‘GeoLocation’column
Itismadeupofaddress,zipcodeandcoordinates"1947E31STST21218(39.32634,-76.587748)“
Wewantjustthosecoordinates
GetData! HowdowemakethisdatareadyforGIS?
a.k.a Pointdata
©Fearless|page38
Todothis,wewillusetwotoolsinthespreadsheet:
• FindandReplace(Ctrl+H)
• TexttoColumns
Convertweirdgeolocationtopointdata
©Fearless|page39
SelectthewholeGeolocationcolumnbyclickingtheheaderrow
Convertweirdgeolocationtopointdata
©Fearless|page40
Openfindandreplace
Convertweirdgeolocationtopointdata
©Fearless|page41
Replacelinebreaks(\n)withasemicolon(;)Thisrequireschecking‘regularexpressions’on.The\nisa‘regularexpression’computercodeforalinebreak.
Whenready,clickReplaceAll.
Convertweirdgeolocationtopointdata
©Fearless|page42
NowtheGeolocationisonasingleline
Convertweirdgeolocationtopointdata
©Fearless|page43
Next,splittheGeolocationcolumnintothreenewcolumnsbybreakingeachrowatthesemicolon(makesuretheGeolocationisselected)
Convertweirdgeolocationtopointdata
SelecttheData>TexttoColumnstool
©Fearless|page44
UnderSeparatorOptions,selectonlythesemicolonseparator.
Weusedsemicolontoavoidanyconfusionbetweentabs,spaces,orcommasthatareintheotherpartsoftheGeolocation.
Thepreviewshouldshowthecoordinatesinanewcolumn.
Convertweirdgeolocationtopointdata
©Fearless|page45
Nowwehave3columns.Nextwearegoingtosplitthe3rd columnwhichcontainsthelatitudeandlongitudecoordinatesintotwomorecolumns(oneeachforlatitudeandlongitude
SelectthatlastcolumnandselectData>Texttocolumnsagain.
Convertweirdgeolocationtopointdata
©Fearless|page46
Thistimewewillsplitbasedonthecommatogettwonewcolumns.
Convertweirdgeolocationtopointdata
©Fearless|page47
Highlightthecolumnwiththe(39….NumbersandclickFindandReplace.
Convertweirdgeolocationtopointdata
©Fearless|page48
Almostdone,setuptheFind&ReplacetoFindthesymbol(andreplaceitwithnothing(leavereplaceblank).ThistimeweturnoffRegularexpressionsbecausewecareabouttheactual(character.
Convertweirdgeolocationtopointdata
©Fearless|page49
Tada!Callthisnewcolumnlatitude
Convertweirdgeolocationtopointdata
©Fearless|page50
Nowthistimewiththelastcolumnwiththe-76…numbers,againselectthecolumnandthengotoEdit>Find&Replace
Convertweirdgeolocationtopointdata
©Fearless|page51
Nowreplace)withnothingasbefore
Convertweirdgeolocationtopointdata
©Fearless|page52
Phew,alldone!Callthenewcolumnlongitude.
Youcandeletetheextraleftovercolumnwiththezipcodes(Q)aswedon’tneeditanymore.
SavethisnewfileasaCSV.
NowitstimetobringitintoGIS!
Convertweirdgeolocationtopointdata
©Fearless|page53
BackinQGIS,clicktheiconthatlookslikeacommafor‘AddDelimitedTextLayer’
YoucanalsoaccessthisfromLayer>AddLayer>AddDelimitedTextLayer
Addthe311tabletothemap
©Fearless|page54
UseBrowsetoselecttheCSVfilewejustcreated.
InFileFormat,checktheCSVbox,thewindowshouldautomaticallyreadourfileandrecognizethecoordinatefieldslatitudeandlongitude.
Leaveeverythingelseasdefault.ClickOK.
Addthe311tabletothemap
©Fearless|page55
AnewwindowpopsupaskingfortheCoordinatesystemreferenceforthe311layer.
SelectWGS84
Coordinateslikethis:39.315295,-76.57893AretypicallycalledgeographicorWGScoordinatesandareacommonGPSlatitude,longitudeformat.
Addthe311tabletothemap
©Fearless|page56
Projections/coordinatesystemsareaveryimportantpartofGIS.IfGISfilesarebooksindifferentlanguagesandtheGISsoftwareisatranslator,thentheprojectionstellthetranslatorwhatlanguagethebookwaswritteninsothatitcandoitsjoboftranslating.
Coordinateslikethis:39.315295,-76.57893aretypicallycalledgeographicorWGScoordinatesandareacommonGPSlatitude,longitudeformat.Theircoordinatesystem(projection)languageisWGS84(WorldGeodeticSystem1984).BytellingtheGISsoftwarethatour311coordinatesareoftheWGS84coordinatesystem,itwillknowhowtocorrectlytranslateorpositionthosecoordinatesonamap.
Sometakeawaysonprojections:1. Theyarealwaysneeded2. Ifworkingwithmultipledatalayersyoushouldstrivetogetallyourlayerstobethe
samecoordinatesystemthrough‘re-projecting’thelayers.
QGIShasgreatinformationonprojections:https://docs.qgis.org/2.2/en/docs/user_manual/working_with_projections/working_with_projections.html
Projections!!!
©Fearless|page57
Yeah!Nowour311Ratruboutpointsareonthemap!
Addthe311tabletothemap
©Fearless|page58
Beforewegoon,weneedtosaveour311pointsasashapefilesothatQGISrememberswhatitsprojectionisandsothatifwewanttocomebacklaterandusethisdata,wedon’thavetoreloaditfromcsv.
RightclickthepointlayerintheLayersPanelandselectSaveAs.
Savethe311dataasaShapefile
©Fearless|page59
ClickBrowsenexttoSaveAsandspecifyafilenameandalocationforsavingthenewfile.Iused:311_Customer_Service_Requests_WGS84.shp
LeaveeverythingelsethesameandclickOK.
Savethe311dataaShapefile
©Fearless|page60
RightClickontheoriginal311pointslayerandselectremovetoremoveitfromthemap.
Savethe311dataaShapefile
©Fearless|page61
Nowweshouldsetthewholemap(thepartofQGISthatwearelookingat)tothesameprojection.Atthebottom-leftoftheQGISwindowtherewillbesometextwithalittleglobenexttoit,thistellsusthecurrentmapprojection,wewantthistobethesameasourWGS84shapefiles.Clickthetextnexttotheglobe.
Settheoverallmapprojection
©Fearless|page62
SelecttheWGS84projectionandclickOK.Themapshouldn’tchange,butthetextinthecornerwillnowsayEPSG:4326,whichistheprojectioncodeforWGS84,youmighthavetosearchforit.
Settheoverallmapprojection
The(OTF)meansthaton-the-flyprojectionsareenabled,whichallowsQGIStoautomaticallytranslatebetweendifferentprojections.
©Fearless|page63
NextwearegoingtousetheVector>AnalysisTools>PointsinPolygontooltoperforma‘Spatialjoin’countinghowmany311onepointsfallinsideeachcensustract.
Combiningthetractsand311points
©Fearless|page64
Inthewindowthatpopsupwewanttoleaveeverythingasdefault.TheInputpolygonvectorlayeristhecensustracts.Thepointvectorlayeristhe311pointsandwewanttouseSUMtoaggregatethedata.
Atthebottom,clickBrowse,andspecifyanoutputfilenameandlocation,clickOK.
Combiningthetractsand311points
©Fearless|page65
Uhoh!ACRSWarning!Thetwolayersarenotinthesamecoordinatesystemandaspatialjoinisnotagoodidea!
ClickOK.ItturnsoutQGISwilllettheactiontakeplaceanyway,makingthenewlayer,butwewanttofixthisCRSproblemsincewedon’twantbadresults.
CloseCountPointsinPolygon.
Combiningthetractsand311points
©Fearless|page66
Thenewlayerwejustcreatedwiththedifferentcoordinatesystemsnowshowsupinthe‘LayersPanel’
Let’sremoveitfromthelistandthemapbyrightclickingthenameintheLayersPanelandselectingremovelayer.
ConvertingtheCensusTractLayertoWGS84
Now,rightclicktheinitialcensustractslayerandselect‘SaveAs’
©Fearless|page67
FromthiswindowwearegoingtotellQGIStosaveanewcopyofthecensustractslayer,butthistimewewanttotranslateitandchangeitscoordinatesystemtomatchthepointslayerWGS84.
SpecifyaSaveAsnamebyclickingbrowse,thenselectingalocationandafilename,Inamedmynewfile2010_Census_Profile_by_Census_Tracts_WGS84.shp
ClicktheglobeicontorightofCRS
ConvertingtheCensusTractLayertoWGS84
©Fearless|page68
Thislooksfamiliar!
SelectWGS84.QGISknowstheinitialcoordinatesystemofthecensustractsandbytellingitherewewantittobeinWGS84instead,itwillknowhowtotranslatethecoordinatesfromoneprojectionlanguagetoanother.
ClickOK,thenOKagainonthepreviouswindow.
ConvertingtheCensusTractLayertoWGS84
©Fearless|page69
ThenewWGS84censustractslayerwasaddedtothemap,soletsgoaheadandremovetheinitiallayerbyrightclickingonitandselectingremove.
ConvertingtheCensusTractLayertoWGS84
©Fearless|page70
Nowboththe311pointsandthecensustractsareWGS84shapefilesandthemapisinWGS84.LetsgobacktoPointsinPolygonandtrythatagain.
Combiningthetractsand311points
©Fearless|page71
We’lldoallthesamestepsasbefore,butthistimewhenweclickOK,wewon’tgetaCRSwarning.
Youcanclosethiswindowwhenitsdone.
WecanalsoremovetheotherCensustractslayertoo.
Combiningthetractsand311points
©Fearless|page72
AnalyzingandMappingtheresults
Let’sexploreallourhardwork.Clickthelittleiconwithan‘i’inabluecircle.
Thisistheidentifytool.Clickingonthedifferentcensustractsonthemapwillrevealtheirinformation.
©Fearless|page73
AnalyzingandMappingtheresults
Wecanalsolookattheentireattributetableofthecensustracts.
RightclickthecensustractslayerandselectOpenAttributeTable.
©Fearless|page74
AnalyzingandMappingtheresults
Eachrowisacensustractpolygonandeachcolumnisadifferentattribute.AllthewayattherightisthefieldPNTCNTthatgotaddedwhenwejoinedthe311data.Thisrepresentsthenumberof311pointsineachpolyon.
Letscolorthemapbasedonthisfield!
©Fearless|page75
AnalyzingandMappingtheresults
Closetheattributetable,rightclickonthecensustractlayerintheLayersDialogandselectProperties.
InthewindowthatpopsupselectStyle.
ClickthedropdownthatsaysSingleSymbolandselect‘Graduated’
©Fearless|page76
AnalyzingandMappingtheresults
ChangetheColumntoPNTCNT.Pickacolorrampyoulike.ClickClassify– thewindowwillautomaticallybuildalegend.ClickOKtoseeyournewmap!
©Fearless|page77
AnalyzingandMappingtheresults
Woohoo!That’snotveryexcitingIguess.LooksliketheCitymighthavebeenrespondingtoalotofcallsinthoseneighborhoods.Plusthefirstcategoryis0-78,whichseemslikeitwouldbemisleading!
©Fearless|page78
AnalyzingandMappingtheresults
Let’sadjustthecategoriesabit,leavingonecategoryforjustcensustractsthathadzerocalls,andthenacategoryfortractswithonlyonecall.
©Fearless|page79
AnalyzingandMappingtheresults
That’salittlemoreinteresting,letsmakeamapPDFofthisandshipit!
ButfirstweneedtomakeBaltimorelooknormalandnotallsquishy.
©Fearless|page80
OntheFlyprojection
Before,wesetthelayersandmaptoallthesameprojection,WGS84.Butthatprojectionisjustforgeneraluseanddoesn’tdoagoodjobofpreservingthetrueshapeofthings.Sinceprojectionsarerulesforhowtomaketheroundglobelayflatonapage,somethingisalwayslostordistortedintheprocess.
ButwiththeOntheFlyprojectionmodewecanchangethemaptoabetterprojectionforshowingthetrueshapeofBaltimore,withouthavingtochangeourlayers.
ClickthetextnexttotheglobeagainandthistimeselecttheprojectionwiththeEPSGCode102285thisisforthe“NAD1983HARNMarylandStatePlaneLambertConformalConic”projection!
©Fearless|page81
OntheFlyprojection
Now,thatlooksbetter.
©Fearless|page82
Ctrl+PorFile>NewPrintComposertobringuptheComposerwindow.Giveanamewhenprompted.
Let’sprintthemap
©Fearless|page83
IntheleftpaneloftheComposerwindowselecttheAddNewMapiconandclickanddragasquareontheblankcanvasforwhereyourmapwillbe.
TherearegreatTutorialsforcomposerhere:
http://www.qgistutorials.com/en/docs/making_a_map.html
Let’sprintthemap
©Fearless|page84
UnderLayout,selectLegendanddrawarectangletocreateanewareaforthelegend
Let’sprintthemap
©Fearless|page85
Intherightpanel,selectingItemProperties,thenuncheckingAutoUpdate,willallowyoutoremoveandchangetheitemsthatappearinthelegend.
Let’sprintthemap
©Fearless|page86
Othertoolsallowyoutoaddtextlabels,images,orothermaps.WhenyouaredoneyoucanexportasaPDForanimage.
Let’sprintthemap
©Fearless|page87
• Downloadedspatialandnon-spatialtabledata
• Performeddatatablemanipulationstogetthe311dataintothe
correctformatusingFOSSOfficeSuiteLibreOffice.
• Importedandre-projecteddatainFOSSQGIS
• Performedaspatialjoinbetweenpointandpolygondata
• Styledmapsymbologytobetterreflectthedata
• BuiltasimplemaplayoutandexportedittoaPDF
Summary
©Fearless|page88
• OnlineGISclassesandTutorials:http://open.ems.psu.edu/courseware,http://www.qgistutorials.com/en/
• FOSSSoftwareforGISQGIS(QuantumGIS)http://www.qgis.org/en/site/,LibreOffice (https://www.libreoffice.org/)
• SpatialJoins:http://www.qgistutorials.com/en/docs/performing_spatial_joins.html
• TableJoins:http://www.qgistutorials.com/en/docs/performing_table_joins.html?highlight=table%20join
• Geocoders:https://www.census.gov/geo/maps-data/data/geocoder.htmlhttp://wiki.openstreetmap.org/wiki/Nominatim
• Projections:https://docs.qgis.org/2.2/en/docs/user_manual/working_with_projections/working_with_projections.html
• MapComposer:http://www.qgistutorials.com/en/docs/making_a_map.html
Resources:
©Fearless|page89
Bonus!– show311callsaonawebpage
Iincludedasimplehtmlwebpagewithsomejavascript forshowingour311callsonamap.
The311callswereexportedasGeoJSON inQGIS,thensavedtoavariableinajavascript filerat_311_calls.js.
ThenasimplehtmlpagewassetupwithLeaflettodrawamarkerforeachofthecallsonawebmap.
Youcanaddlotsoffeaturesofyourownincludingcustomiconsandreadingthecalldatawhenyouclickapin:http://leafletjs.com/examples/quick-start/http://leafletjs.com/examples/custom-icons/