building openstreetmap.org, sotm us 2015
TRANSCRIPT
Building OpenStreetMap.org
Tom Hughes
[email protected]@thughes
Session all about the code behind osm.org and how we develop it and how you can helpI'm Tom Hughes and I've been helping run the servers since 2007I work on everything but in particular on the main web site
Web Site Architecture
DatabaseServerWeb/RailsServerWeb/RailsServerWeb/RailsServerRailsServerRailsServerRailsServer
FileServer
Quick overview of architecture**Three frontends using round robin DNSRun apache and passengerHandle most requestsHeavy requests proxied to shared backend pool**Three backendsRun apache, passenger and cgimapSome API requests sent to cgimapEverything else handled by passenger (currently)
Database Schema
Usersusers / user_tokens / user_preferences / user_roles / user_blocks
Geodatachangesets / changeset_tags / changeset_comments / ...
current_nodes / current_node_tags / nodes / node_tags
current_ways / current_way_tags / current_way_nodes /
current_relations / current_relation_tags / current_relation_members /
GPS Datagpx_files / gpx_file_tags / gps_points
Notesnotes / note_comments
Overview of databaseUsers table, with various auxiliary tables**Geodata, everything grouped by changesets**Nodes current tables with latest version, history tables with all versions yes duplicated...**Repeated for ways and relations but with tables to link to child objects**Imported GPS data yes the names are not consistent**Notes with commentsVarious others with diaries, friends, messages,etc
The Rails Port
Main web site is Ruby on Rails known as the rails port for historical reasonsFind it on github issues welcome pull requests even more welcomeIncludes an implementation of the APIAlternate implementation of part of the API is what is used in production though more later
The Rails Port
Getting StartedINSTALL.md
CONFIGURE.md
CONTRIBUTING.md
AlternativelyVAGRANT.md
We have issues...
Get started checkout the code!Do a (small) bit of reading**INSTALL.md tells you how to do the basic setup**CONFIGURE.md covers a few things you may want to do once you have the basics working**CONTRIBUTING.md explains how we work **Or you can use vagrant to do much of the basic setup read VAGRANT.md for more information**If you're looking for things to do then we have open issues!
The Rails Port
Models app/models
Views app/views
Controllers app/controllers
Javascript app/assets/javascripts
Stylesheets app/assets/stylesheets
Strings config/locales/en.yml
Structure is a basic MVC architectureModels represent the data with a class for each type of data**Views are what the user sees we use ERB templates which are a mix of HTML and Ruby**Controllers handle user requests fetching the needed data and then rendering a view**We also have client side javascript code using jQuery**And stylesheets to control client side rendering of the views**All text needs to be translatable add strings to english and Translatewiki does the rest
The Rails Port
ModelsNode class is current_nodes
OldNode class is nodes
Trace class is gpx_files
Tracetag class is gpx_file_tags
Tracepoint class is gps_points
Models have one class per database tableMostly the names matchExcept when they don't...**Big difference is that current tables lose their prefix**and history tables gain an old prefixApplies to ways and relations as well**GPS tables also differentThis is largely a question of history and the rails code being written around an existing schema
The Rails Port
Stylerubocop for ruby
rake jshint for javascript
Testsrake test for ruby
javascript coming soon?
Travis will run on pull requests
Coveralls will check coverage
Please run style checkers we have rubocop setup for ruby and jshint for rails**Even more important please run the testsWe have good tests for the ruby codeBut not (yet) for javascript**Style checks and tests will be run by travis when you open a pull**Coverage will also be checked by coveralls only the ruby code at the moment but coverage is good and we'd like to keep it that waySo add tests for any new ruby code!
The Rails Port
MergingPull requests notified to rails-dev list
Review of behaviour and UI changes
Review of code changes
Larger changes may get a test instance
When your change is ready, open a PRThe rails-dev list will be notified, as it is for all issues**First thing we will usually do is look at the change as a user what are the UI changes etc**When we're happy with that I will usually look at the actual code for any implementation issuesNot hard and fast though, things may happen in a different order, and small fixes are often just merged straight off**Larger changes may get a test instance, with people invited to try it out
CGImap
Second part of our story is CGImapAlso on githubC++ implementation of (part of) the APIIncreased performance by carefully tailored SQL queriesBetter memory management for potentially large data sets
CGImap
Currently handles a number of API callsThe main map call
Single node, way and relation fetches
Multiple node, way and relation fetches
The full way and relation calls
Want to extend it to cover more API callsHistory calls
Changeset calls
Currently handlesThe main map call what editors use to get dataFetching individual map objectsFetching multiple map objectsFetching map objects with their child objects**Want to extend itCalls which include object historyChangeset callsAnd so on
CGImap
Currently runs as a separate daemonAccessed via FastCGI
Core code has now been split into a libraryWill allow creation of a ruby gem around the library
Which then allows direct integration with rails code
JSON output supportCan be compiled with JSON support
Not currently enabled in production
Currently runs as a separate daemon called via FastCGI**Core code has been split into a libraryWill allow a gem wrapper to be written for integration with rails codeAvoids current duplication of API code between rails and cgimap**Supports JSON but not currently enabled as only calls handled by cgimap would be supportedMoving rails code to call cgimap via gem wrapper will allow this (and other changes) to be implemented onceNeed to reach reasonable API completeness in cgimap first though help please!
CGImap
StyleFormatted in LLVM style with clang-format
Configure --with-clang-format
Run make clang-format to reformat
TestsRun tests with make check
Needs postgresql running
Travis coming soon?
Formatted in LLVM style with clang-formatNeed to configure with-clang-format then you can reformat with make clang-formatNeeds clang 3.6**Run the tests with make checkNeed to have postgres runningTravis config merged but not enabled yet
Get Involved
GitHubhttps://github.com/openstreetmap/openstreetmap-websitehttps://github.com/zerebubuth/openstreetmap-cgimapMailing Listshttps://lists.openstreetmap.org/listinfo/rails-devhttps://lists.openstreetmap.org/listinfo/devIRC#osm-devEngineering Working Grouphttps://wiki.osmfoundation.org/wiki/[email protected]
None of this happens by magic, and many of the people working on this are also trying to keep the servers runningSo please, get involved if you're planning something big, or just want advice, then please contact us and we'll be glad to helpI can take some questions now, or find me later I'm around all weekend and for the hackday on MondayAnd if you want to know more about how we actually turn all this code into a running service then stay around for my colleague Grant Slater's talk which is up next in this room