practical data visualization · overview infrastructure ... feature dev test github pull request...
TRANSCRIPT
![Page 1: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/1.jpg)
Practical Data Visualization Infrastructure, Tools, and Project Management
Jonathan Page
![Page 2: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/2.jpg)
Overview
▪ Infrastructure
▪ Development and Monitoring Tools
▪ Project Management
![Page 3: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/3.jpg)
Infrastructure Virtual Machines and Server Applications
![Page 4: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/4.jpg)
Production
Database
User perspective
![Page 5: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/5.jpg)
Production
Database
User perspective
![Page 6: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/6.jpg)
Production
File Database
Worker
3rd Party Perspective
New
Data Files
![Page 7: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/7.jpg)
Production
File Database
Admin Perspective
New
Data Source
![Page 8: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/8.jpg)
Production Staging
File Deployment Database
Worker
Nightly Data Update
Worker
![Page 9: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/9.jpg)
Production Staging
Deployment Database
Worker
Developer Perspective New
Feature
Dev Test
GitHub Pull Request
Travis CI
Approval
![Page 10: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/10.jpg)
Production Server
▪ UHERO Data Manager (Ruby on Rails)
▪ REST API (Go)
▪ NGINX – serves static files
▪ UHERO Data Portal (Angular, Highcharts)
▪ DBEDT Data Warehouse (Angular)
![Page 11: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/11.jpg)
Database Server
▪ MySQL
▪ Redis
![Page 12: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/12.jpg)
Worker Server
▪ Sidekiq Workers (Ruby on Rails)
![Page 13: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/13.jpg)
File Server
▪ SMB File Server
▪ Stores raw data files for future processing and backup
![Page 14: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/14.jpg)
Staging Server
▪ Sandbox for testing code/DB changes before move to production
▪ All applications on Production Server
▪ Nightly copy of DBs from Database Server
▪ Testing subset of File Server contents
▪ Sidekiq workers from Worker Server
![Page 15: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/15.jpg)
Open Source Software
▪ github.com/UHERO/udaman
▪ github.com/UHERO/rest-api
▪ github.com/UHERO/data-portal
▪ github.com/UHERO/dbedt-data-portal
![Page 16: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/16.jpg)
Lessons Learned
▪ AWS (the cloud in general) is more expensive than hosting with University ITS VMs
▪ Prefer a small group of well chosen programming languages
▪ Building is more fun than supporting, so build carefully
![Page 17: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/17.jpg)
Tools Development and Monitoring
![Page 18: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/18.jpg)
Development Tools
▪ Development Environments
▪ IDEs: RubyMine, WebStorm, SublimeText, Atom, Vim
▪ Vagrant, [Future] Docker
▪ Version Control
▪ Git, GitHub.com
▪ Code Review
▪ GitHub Pull requests
![Page 19: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/19.jpg)
Development Tools (continued)
▪ Testing
▪ Travis.CI
▪ Deployment
▪ Currently manual runbook, [Future] Bamboo (Atlassian)
▪ Documentation
▪ Confluence (Atlassian)
▪ Issue Management
▪ JIRA (Atlassian)
![Page 20: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/20.jpg)
Lessons Learned
▪ Stick to industry standard tools as much as possible (community help is invaluable)
▪ There is always a newer, shinier tool; “if it aint broke, don’t fix it”
▪ Some tools allow you to build more reliable and more maintainable software
▪ Tool switching is not costless
![Page 21: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/21.jpg)
Project Management Agile
![Page 22: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/22.jpg)
Agile
▪ Kanban for service teams
▪ Scrum for development teams
▪ JIRA (Atlassian)
![Page 23: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/23.jpg)
Scrum Ceremonies
▪ Daily Scrum (Standup)
▪ Sprint Planning (Story Grooming, Planning Poker)
▪ Showcase (Sprint Review)
▪ Sprint Retrospective
![Page 24: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/24.jpg)
Lessons Learned
▪ Scrum ceremony adherence should depend on development team size
▪ If the structure of Scrum is intimidating/overbearing, try Kanban
▪ If Kanban feels to unstructured, try Scrum
▪ It’s not uncommon for development teams in the tech industry to switch between flavors of Agile
▪ See if your University ITS has a license for JIRA
![Page 25: Practical Data Visualization · Overview Infrastructure ... Feature Dev Test GitHub Pull Request Travis CI Approval . Production Server ... Currently manual runbook, [Future] Bamboo](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f2af99c2046d9645970f6ca/html5/thumbnails/25.jpg)