free and open source gis technology workshop - …€¦ · free and open source gis technology...

BaltimoreDataDay2017BaltimoreNeighborhood

IndicatorsAlliance

FreeandOpenSourceGISTechnologyWorkshopJonathanDandois,Ph.D.

14July2017

Preparedfor:

©Fearless|page2

Background

WhatisGIS?

GeographicInformationSystems

• GISisthetechnologythathelpscombinedifferenttypesofdatatohelpexplore,analyze,andcommunicatespatialpatterns.

• WithGIS,weoftencombinespatialandnon-spatialdata(tables),andstackmultiplelayersofdatatorepresenttheworld

https://www.in.gov/gis/gis101.htm

©Fearless|page3

WhatcanyoudowithGIS?

AnalyzeandCommunicate

• Analyze->Answercomplexquestionsinvolvingspaceandtime:

• Howmanypeoplelivewithin1mileofaparkandhowlonghavetheylivedthere?

• Communicate->makemapstodeliveramessage

• Howdowechoosemapstylestoconveyourmessage:colors,linethickness,transparency,patterns,symbols?

https://www.in.gov/gis/gis101.htm

©Fearless|page4

TypesofGISData

SpatialData– VectorData

Polygons:Neighborhoods,States,Counties,CensusTracts

Points:Events,locations,stores

Lines:Roads,streams,routes

©Fearless|page5

TypesofGISData

Non-SpatialTabularData

DatatablesfromCSVs,Excel,etc.

©Fearless|page6

TypesofGISData

Non-SpatialTabularData

DatatablesfromCSVs,Excel,etc.

WhatinformationinadatatablecanweusetoRELATEtoourmapdata?

©Fearless|page7

AdditionalResourcesforintrotoGIS

• PennStateUniversityOpenCourses (http://open.ems.psu.edu/courseware)

• Basic:Geography160:MappingourChangingWorld(https://www.e-education.psu.edu/geog160/)

• Complex:GEOG585:OpenWebMapping(https://www.e-education.psu.edu/geog585/)

• QGIS(QuantumGIS)Tutorialshttp://www.qgistutorials.com/en/

©Fearless|page8

ToolsandSoftware

MainTools

GISSoftware

• QGIS(QuantumGIS)http://www.qgis.org/en/site/

• DesktopGISApplicationformapping,cartography,analysis

OfficeTools(somethingtomanagespreadsheetsandtables)

• LibreOffice (https://www.libreoffice.org/)

• FreeandopensourceOffice

©Fearless|page9

ToolsandSoftware

OtherTools

• OSGeo ‘Suiteofgeotools’http://www.osgeo.org/

• GRASSGIS– GISformorecomplexanalyses

• Geoserver /MapServer forsharinglotsofdataontheweb

• OpenStreetMap (mapwebsite,downloadbasemaps,useopengeotoolslikeageocoder)

• ESRIsuiteoftools– usewhatyouhave.

©Fearless|page10

EXAMPLE!CookingClass

Whereare311callsforrats?

Combine311tabledatawithcensustracts.Mapby:‘callcount’pertract.

©Fearless|page11

Getthingssetup

Installprograms

GetandinstallQGISandLibreOffice

QGIS(QuantumGIS)http://www.qgis.org/en/site/

LibreOffice https://www.libreoffice.org/

©Fearless|page12

GetData!

Whatformatsofdataarewelookingfor?

SpatialData

• Shapefile(SHP)- oldformatfromESRI,usuallycomesasazipcomposedofmanyfiles

• GeoJSON (GeoJavascriptObjectNotation)allthedataisstoredinasingletextfile

Others(TopoJSON,geodatabase…)

©Fearless|page13

GetData!

Whatformatsofdataarewelookingfor?

TableData

• CSV(commaseparatedvalues)

• Excelspreadsheets

• TXTtextfile

©Fearless|page14

GetData!

CensusTracts

StartatBaltimoreCityOpenData

1. https://data.baltimorecity.gov/

2. Searchforcensustracts

©Fearless|page15

GetData!

CensusTracts

©Fearless|page16

GetData!

Unzipthefile

Allofthesefiles(exceptforthezip)makeuptheshapefile!

©Fearless|page17

LoadthecensustractsintoQGIS

©Fearless|page18


©Fearless|page19


©Fearless|page20


©Fearless|page21


Onlyselectthe.shp file,clickOpen.

©Fearless|page22


©Fearless|page23


©Fearless|page24


©Fearless|page25

GetData!

311Data

https://data.baltimorecity.gov/City-Services/311-Customer-Service-Requests/9agw-sxsr/data

©Fearless|page26

GetData!

Startbyapplyingsomefilterstogetthedatawewant

©Fearless|page27

GetData!

©Fearless|page28

GetData! AddfilterforSRType contains‘rat’

©Fearless|page29

GetData! AddfilterforCreatedDate after7/7/2017

©Fearless|page30

GetData! Download/Exportthedata

©Fearless|page31

GetData! LoadCSVintoLibreOffice

©Fearless|page32


©Fearless|page33


©Fearless|page34


©Fearless|page35


©Fearless|page36

Wehaveneighborhoodinformation:

SoifwehaveamapofNeighborhoodnames,wecouldtrya‘TableJoin’http://www.qgistutorials.com/en/docs/performing_table_joins.html?highlight=table%20join

Wehaveaddresses,sowecould‘geocode’(assignmaplatitudeandlongitudecoordinatestoaddresses:https://www.census.gov/geo/maps-data/data/geocoder.htmlhttp://wiki.openstreetmap.org/wiki/Nominatim

GetData! HowdowemakethisdatareadyforGIS?

©Fearless|page37

Butwealsohavea‘GeoLocation’column

Itismadeupofaddress,zipcodeandcoordinates"1947E31STST21218(39.32634,-76.587748)“

Wewantjustthosecoordinates

GetData! HowdowemakethisdatareadyforGIS?

a.k.a Pointdata

©Fearless|page38

Todothis,wewillusetwotoolsinthespreadsheet:

• FindandReplace(Ctrl+H)

• TexttoColumns

Convertweirdgeolocationtopointdata

©Fearless|page39

SelectthewholeGeolocationcolumnbyclickingtheheaderrow


©Fearless|page40

Openfindandreplace


©Fearless|page41

Replacelinebreaks(\n)withasemicolon(;)Thisrequireschecking‘regularexpressions’on.The\nisa‘regularexpression’computercodeforalinebreak.

Whenready,clickReplaceAll.


©Fearless|page42

NowtheGeolocationisonasingleline


©Fearless|page43

Next,splittheGeolocationcolumnintothreenewcolumnsbybreakingeachrowatthesemicolon(makesuretheGeolocationisselected)


SelecttheData>TexttoColumnstool

©Fearless|page44

UnderSeparatorOptions,selectonlythesemicolonseparator.

Weusedsemicolontoavoidanyconfusionbetweentabs,spaces,orcommasthatareintheotherpartsoftheGeolocation.

Thepreviewshouldshowthecoordinatesinanewcolumn.


©Fearless|page45

Nowwehave3columns.Nextwearegoingtosplitthe3rd columnwhichcontainsthelatitudeandlongitudecoordinatesintotwomorecolumns(oneeachforlatitudeandlongitude

SelectthatlastcolumnandselectData>Texttocolumnsagain.


©Fearless|page46

Thistimewewillsplitbasedonthecommatogettwonewcolumns.


©Fearless|page47

Highlightthecolumnwiththe(39….NumbersandclickFindandReplace.


©Fearless|page48

Almostdone,setuptheFind&ReplacetoFindthesymbol(andreplaceitwithnothing(leavereplaceblank).ThistimeweturnoffRegularexpressionsbecausewecareabouttheactual(character.


©Fearless|page49

Tada!Callthisnewcolumnlatitude


©Fearless|page50

Nowthistimewiththelastcolumnwiththe-76…numbers,againselectthecolumnandthengotoEdit>Find&Replace


©Fearless|page51

Nowreplace)withnothingasbefore


©Fearless|page52

Phew,alldone!Callthenewcolumnlongitude.

Youcandeletetheextraleftovercolumnwiththezipcodes(Q)aswedon’tneeditanymore.

SavethisnewfileasaCSV.

NowitstimetobringitintoGIS!


©Fearless|page53

BackinQGIS,clicktheiconthatlookslikeacommafor‘AddDelimitedTextLayer’

YoucanalsoaccessthisfromLayer>AddLayer>AddDelimitedTextLayer

Addthe311tabletothemap

©Fearless|page54

UseBrowsetoselecttheCSVfilewejustcreated.

InFileFormat,checktheCSVbox,thewindowshouldautomaticallyreadourfileandrecognizethecoordinatefieldslatitudeandlongitude.

Leaveeverythingelseasdefault.ClickOK.


©Fearless|page55

AnewwindowpopsupaskingfortheCoordinatesystemreferenceforthe311layer.

SelectWGS84

Coordinateslikethis:39.315295,-76.57893AretypicallycalledgeographicorWGScoordinatesandareacommonGPSlatitude,longitudeformat.


©Fearless|page56

Projections/coordinatesystemsareaveryimportantpartofGIS.IfGISfilesarebooksindifferentlanguagesandtheGISsoftwareisatranslator,thentheprojectionstellthetranslatorwhatlanguagethebookwaswritteninsothatitcandoitsjoboftranslating.

Coordinateslikethis:39.315295,-76.57893aretypicallycalledgeographicorWGScoordinatesandareacommonGPSlatitude,longitudeformat.Theircoordinatesystem(projection)languageisWGS84(WorldGeodeticSystem1984).BytellingtheGISsoftwarethatour311coordinatesareoftheWGS84coordinatesystem,itwillknowhowtocorrectlytranslateorpositionthosecoordinatesonamap.

Sometakeawaysonprojections:1. Theyarealwaysneeded2. Ifworkingwithmultipledatalayersyoushouldstrivetogetallyourlayerstobethe

samecoordinatesystemthrough‘re-projecting’thelayers.

QGIShasgreatinformationonprojections:https://docs.qgis.org/2.2/en/docs/user_manual/working_with_projections/working_with_projections.html

Projections!!!

©Fearless|page57

Yeah!Nowour311Ratruboutpointsareonthemap!


©Fearless|page58

Beforewegoon,weneedtosaveour311pointsasashapefilesothatQGISrememberswhatitsprojectionisandsothatifwewanttocomebacklaterandusethisdata,wedon’thavetoreloaditfromcsv.

RightclickthepointlayerintheLayersPanelandselectSaveAs.

Savethe311dataasaShapefile

©Fearless|page59

ClickBrowsenexttoSaveAsandspecifyafilenameandalocationforsavingthenewfile.Iused:311_Customer_Service_Requests_WGS84.shp

LeaveeverythingelsethesameandclickOK.

Savethe311dataaShapefile

©Fearless|page60

RightClickontheoriginal311pointslayerandselectremovetoremoveitfromthemap.

Savethe311dataaShapefile

©Fearless|page61

Nowweshouldsetthewholemap(thepartofQGISthatwearelookingat)tothesameprojection.Atthebottom-leftoftheQGISwindowtherewillbesometextwithalittleglobenexttoit,thistellsusthecurrentmapprojection,wewantthistobethesameasourWGS84shapefiles.Clickthetextnexttotheglobe.

Settheoverallmapprojection

©Fearless|page62

SelecttheWGS84projectionandclickOK.Themapshouldn’tchange,butthetextinthecornerwillnowsayEPSG:4326,whichistheprojectioncodeforWGS84,youmighthavetosearchforit.

Settheoverallmapprojection

The(OTF)meansthaton-the-flyprojectionsareenabled,whichallowsQGIStoautomaticallytranslatebetweendifferentprojections.

©Fearless|page63

NextwearegoingtousetheVector>AnalysisTools>PointsinPolygontooltoperforma‘Spatialjoin’countinghowmany311onepointsfallinsideeachcensustract.

Combiningthetractsand311points

©Fearless|page64

Inthewindowthatpopsupwewanttoleaveeverythingasdefault.TheInputpolygonvectorlayeristhecensustracts.Thepointvectorlayeristhe311pointsandwewanttouseSUMtoaggregatethedata.

Atthebottom,clickBrowse,andspecifyanoutputfilenameandlocation,clickOK.


©Fearless|page65

Uhoh!ACRSWarning!Thetwolayersarenotinthesamecoordinatesystemandaspatialjoinisnotagoodidea!

ClickOK.ItturnsoutQGISwilllettheactiontakeplaceanyway,makingthenewlayer,butwewanttofixthisCRSproblemsincewedon’twantbadresults.

CloseCountPointsinPolygon.


©Fearless|page66

Thenewlayerwejustcreatedwiththedifferentcoordinatesystemsnowshowsupinthe‘LayersPanel’

Let’sremoveitfromthelistandthemapbyrightclickingthenameintheLayersPanelandselectingremovelayer.

ConvertingtheCensusTractLayertoWGS84

Now,rightclicktheinitialcensustractslayerandselect‘SaveAs’

©Fearless|page67

FromthiswindowwearegoingtotellQGIStosaveanewcopyofthecensustractslayer,butthistimewewanttotranslateitandchangeitscoordinatesystemtomatchthepointslayerWGS84.

SpecifyaSaveAsnamebyclickingbrowse,thenselectingalocationandafilename,Inamedmynewfile2010_Census_Profile_by_Census_Tracts_WGS84.shp

ClicktheglobeicontorightofCRS


©Fearless|page68

Thislooksfamiliar!

SelectWGS84.QGISknowstheinitialcoordinatesystemofthecensustractsandbytellingitherewewantittobeinWGS84instead,itwillknowhowtotranslatethecoordinatesfromoneprojectionlanguagetoanother.

ClickOK,thenOKagainonthepreviouswindow.


©Fearless|page69

ThenewWGS84censustractslayerwasaddedtothemap,soletsgoaheadandremovetheinitiallayerbyrightclickingonitandselectingremove.


©Fearless|page70

Nowboththe311pointsandthecensustractsareWGS84shapefilesandthemapisinWGS84.LetsgobacktoPointsinPolygonandtrythatagain.


©Fearless|page71

We’lldoallthesamestepsasbefore,butthistimewhenweclickOK,wewon’tgetaCRSwarning.

Youcanclosethiswindowwhenitsdone.

WecanalsoremovetheotherCensustractslayertoo.


©Fearless|page72

AnalyzingandMappingtheresults

Let’sexploreallourhardwork.Clickthelittleiconwithan‘i’inabluecircle.

Thisistheidentifytool.Clickingonthedifferentcensustractsonthemapwillrevealtheirinformation.

©Fearless|page73


Wecanalsolookattheentireattributetableofthecensustracts.

RightclickthecensustractslayerandselectOpenAttributeTable.

©Fearless|page74


Eachrowisacensustractpolygonandeachcolumnisadifferentattribute.AllthewayattherightisthefieldPNTCNTthatgotaddedwhenwejoinedthe311data.Thisrepresentsthenumberof311pointsineachpolyon.

Letscolorthemapbasedonthisfield!

©Fearless|page75


Closetheattributetable,rightclickonthecensustractlayerintheLayersDialogandselectProperties.

InthewindowthatpopsupselectStyle.

ClickthedropdownthatsaysSingleSymbolandselect‘Graduated’

©Fearless|page76


ChangetheColumntoPNTCNT.Pickacolorrampyoulike.ClickClassify– thewindowwillautomaticallybuildalegend.ClickOKtoseeyournewmap!

©Fearless|page77


Woohoo!That’snotveryexcitingIguess.LooksliketheCitymighthavebeenrespondingtoalotofcallsinthoseneighborhoods.Plusthefirstcategoryis0-78,whichseemslikeitwouldbemisleading!

©Fearless|page78


Let’sadjustthecategoriesabit,leavingonecategoryforjustcensustractsthathadzerocalls,andthenacategoryfortractswithonlyonecall.

©Fearless|page79


That’salittlemoreinteresting,letsmakeamapPDFofthisandshipit!

ButfirstweneedtomakeBaltimorelooknormalandnotallsquishy.

©Fearless|page80

OntheFlyprojection

Before,wesetthelayersandmaptoallthesameprojection,WGS84.Butthatprojectionisjustforgeneraluseanddoesn’tdoagoodjobofpreservingthetrueshapeofthings.Sinceprojectionsarerulesforhowtomaketheroundglobelayflatonapage,somethingisalwayslostordistortedintheprocess.

ButwiththeOntheFlyprojectionmodewecanchangethemaptoabetterprojectionforshowingthetrueshapeofBaltimore,withouthavingtochangeourlayers.

ClickthetextnexttotheglobeagainandthistimeselecttheprojectionwiththeEPSGCode102285thisisforthe“NAD1983HARNMarylandStatePlaneLambertConformalConic”projection!

©Fearless|page83

IntheleftpaneloftheComposerwindowselecttheAddNewMapiconandclickanddragasquareontheblankcanvasforwhereyourmapwillbe.

TherearegreatTutorialsforcomposerhere:

http://www.qgistutorials.com/en/docs/making_a_map.html

Let’sprintthemap

©Fearless|page85

Intherightpanel,selectingItemProperties,thenuncheckingAutoUpdate,willallowyoutoremoveandchangetheitemsthatappearinthelegend.

Let’sprintthemap

©Fearless|page86

Othertoolsallowyoutoaddtextlabels,images,orothermaps.WhenyouaredoneyoucanexportasaPDForanimage.

Let’sprintthemap

©Fearless|page87

• Downloadedspatialandnon-spatialtabledata

• Performeddatatablemanipulationstogetthe311dataintothe

correctformatusingFOSSOfficeSuiteLibreOffice.

• Importedandre-projecteddatainFOSSQGIS

• Performedaspatialjoinbetweenpointandpolygondata

• Styledmapsymbologytobetterreflectthedata

• BuiltasimplemaplayoutandexportedittoaPDF

Summary

©Fearless|page88

• OnlineGISclassesandTutorials:http://open.ems.psu.edu/courseware,http://www.qgistutorials.com/en/

• FOSSSoftwareforGISQGIS(QuantumGIS)http://www.qgis.org/en/site/,LibreOffice (https://www.libreoffice.org/)

• SpatialJoins:http://www.qgistutorials.com/en/docs/performing_spatial_joins.html

• TableJoins:http://www.qgistutorials.com/en/docs/performing_table_joins.html?highlight=table%20join

• Geocoders:https://www.census.gov/geo/maps-data/data/geocoder.htmlhttp://wiki.openstreetmap.org/wiki/Nominatim

• Projections:https://docs.qgis.org/2.2/en/docs/user_manual/working_with_projections/working_with_projections.html

• MapComposer:http://www.qgistutorials.com/en/docs/making_a_map.html

Resources:

©Fearless|page89

Bonus!– show311callsaonawebpage

Iincludedasimplehtmlwebpagewithsomejavascript forshowingour311callsonamap.

The311callswereexportedasGeoJSON inQGIS,thensavedtoavariableinajavascript filerat_311_calls.js.

ThenasimplehtmlpagewassetupwithLeaflettodrawamarkerforeachofthecallsonawebmap.

Youcanaddlotsoffeaturesofyourownincludingcustomiconsandreadingthecalldatawhenyouclickapin:http://leafletjs.com/examples/quick-start/http://leafletjs.com/examples/custom-icons/

free and open source gis technology workshop - …€¦ · free and open source gis technology...

Documents