overcoming gravity & achieving escape velocity · overcoming gravity & achieving escape...
TRANSCRIPT
Overcoming gravity & achieving escape velocity
Aaaaaaaaaaaaaaa
aa
Martin Monkman
Seattle2018-11-07
Presenter
Presentation Notes
<!— Copyright 2018 Province of British Columbia This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. -->
1
Presenter
Presentation Notes
About me… …I have a BSc and MA in Geography. I started writing code in my first year as an undergraduate student in a computer science class, programming FORTRAN on punch cards.
2
Presenter
Presentation Notes
British Columbia is a province in Canada; the 3rd largest in area (944,000 sq km) and population (4.8 million). 47 times the size of Wales.
3
Presenter
Presentation Notes
I live in the provincial capital city, Victoria, which is on the southern end of Vancouver Island. (The island is 1.5 times the size of Wales.) It’s only a 25 minute flight from Seattle.
4
BC Stats
Presenter
Presentation Notes
BC Stats, the provincial statistics agency in British Columbia, got its start in 1894 with the passing of the Legislative Library and Statistics Bureau Act. We provide statistical evidence to policy makers across the government, the broader public sector (such as health administrators and school boards), businesses, and the public. We do this through analysis of existing Statistics Canada data (including the Census of Canada), administrative data collected by government agencies, and surveys that we carry out.
5
What we do
“…collect, compile, compare, analyze, abstract, project and distribute statistical information respecting the commercial, industrial, financial, social, economic and general activities and conditions of British Columbia and persons in British Columbia or doing business in British Columbia”
Presenter
Presentation Notes
British Columbia Statistics Act, section 5 http://www.bclaws.ca/civix/document/id/complete/statreg/96439_01#section5
6
How we do it
Collect
Presenter
Presentation Notes
This model looks very much like what we do at BC Stats … with the possible exception that we are also involved in collecting the data before we can import it. I will talk about some specific examples a bit later on in this presentation. (source: Grolemund & Wickham, R for Data Science https://r4ds.had.co.nz/)
7
Presenter
Presentation Notes
At BC Stats we’d been using a variety of tools including SPSS, Excel, Access, and a few other tools (including APL) to carry out our work. But none of these tools hit all the marks for open, reproducible research Data in proprietary formats No code sharing GUI (no scripting) Costly license fees
8
Presenter
Presentation Notes
But then roughly 8-10 years ago I’d started to hear about these open source tools that are revolutionizing statistical analysis and this new-fangled “data science” thing, So I was curious if R would be something useful for us in our work. In 2012 I set out to learn more, first by downloading and installing R on my work computer.
9
Presenter
Presentation Notes
Only to be met by something like this… And this was common across the BC Government. Anyone attempting to install R and the associated packages would have been met with the same lack of administrative priviledge. To get software installed to a request to our local IT department, who had to request permission to open a 30 minute window to install software. And in my naivety of the open source / multi package model, I didn’t realize that I’d discover packages that I didn’t have installed that would solve the next problem So I was frustrated.
10
Presenter
Presentation Notes
Very frustrating indeed.
11
bayesball.blogspot.com
Presenter
Presentation Notes
So I went home, installed R on my own computer and started learning, writing blogposts of my adventures, including contributing to the Lahman baseball data package But this just made me more frustrated with the policies and procedures of government, as I started to realize that R was just the thing we needed
12
Presenter
Presentation Notes
https://twitter.com/AdamGruer/status/1013538660132589568�
13
Presenter
Presentation Notes
I’m not alone in my frustration with the bureaucracy…it’s a regular feature of popular culture. Here we have the Vogons from The Hitchhiker’s Guide to the Galaxy, - described as "Not actually evil, but bad-tempered, bureaucratic, officious and callous."� but the Hairball is also seen in the Dilbert character Mordac, the Preventer of IT Services Many of the plot lines in Parks and Recreation Image: the Vogons in Hitchhiker’s Guide to the Galaxy.
14
Presenter
Presentation Notes
https://twitter.com/estherderby/status/999367593625612293
15
Presenter
Presentation Notes
I would like to introduce you to a book that gives us a great metaphor for that collection of policies and procedures. The Hairball accumulates. All organizations have them. They are a natural consequence of a legitimate desire to reduce risk and the minimize error. But in total, and when rigidly imposed, they can stifle creativity and innovation. The gravity of the Hairball "exercises an inexorable pull into mediocrity"[^2]. Because it can be exhausting to deal with the Hairball (whether that's working with it, around it, or against it), the very presence of the Hairball can act as an inhibitor. As Mackenzie says, it has an impact on how people approach their work. �
16
Presenter
Presentation Notes
But Mackenzie points out, through some great stories drawn from his personal experience, that it’s possible to go into orbit around the hairball.
17
Life in orbit
Presenter
Presentation Notes
So five years ago, our R implementation was bogged down in the gravity of the Hairball…but now we’re in orbit. Here’s some examples…
18
Presenter
Presentation Notes
https://twitter.com/JennyBryan/status/954251663086256128 To which I replied, “And another 29 @github repos in the not-ready-for-primetime corporate space...” Note: as of 2018-10-25, the count is now 334 in github/bcgov and BC Stats alone has 5 in the beta/dev corporate space.
19
Presenter
Presentation Notes
Then Hadley Wickham made this twitter request…
20
Presenter
Presentation Notes
…to which my colleague Dr Stephanie Hazlitt in the Ministry of Environment’s State of Environment Reporting team replied…
21
Presenter
Presentation Notes
…and the next thing we know, this is incorporated into Hadley Wickham’s “You Can’t Do Data Science in a GUI” presentation! https://speakerdeck.com/hadley/you-cant-do-data-science-in-a-gui
22
Presenter
Presentation Notes
You’ve all have seen the brilliant hex sticker map of Australia that was created for the 2018 useR! conference … two of those are BC Government package hex stickers, including …
23
Presenter
Presentation Notes
… bcmaps https://github.com/bcgov/bcmaps An R package of spatial map layers for British Columbia
24
Presenter
Presentation Notes
BC Stats examples
25
Presenter
Presentation Notes
VLQ – used bookdown start Julie Hawkins presentation https://github.com/bcgov/bcgov-useR/tree/master/2018/jhawkins_case_study
26
Presenter
Presentation Notes
VLQ – used bookdown start Julie Hawkins presentation https://github.com/bcgov/bcgov-useR/tree/master/2018/jhawkins_case_study
27
Employee survey
28
`xlsx`
Presenter
Presentation Notes
Employee surveys – multiple questions, multiple demographic groups Number of respondents and demographic categories varies by specific agency or organization Generation of formatted multi-sheet Excel files using xlsx
29
Post-secondary student outcomes
Presenter
Presentation Notes
http://outcomes.bcstats.gov.bc.ca/Default/Home.aspx Surveys of thousands of former post-secondary students across British Columbia. They are asked a series of questions related to their educational experiences, subsequent employment and further studies.
30
Presenter
Presentation Notes
Outcomes https://github.com/bcgov/student-outcomes-dataviz Housing https://github.com/bcgov/housing-data-challenge-ruby
31
The Launch Pad
Presenter
Presentation Notes
The launch pad
32
Presenter
Presentation Notes
So I sought out information about the Administrative Privilege Policy for our ministry, thinking that there might be a way to make a request to have it granted. I found this rather lengthy policy document: You may note that in the first paragraph the reader is referred to a second policy document, the BC Government's Information Security Policy. The Hairball!
DataBC catalogue
33
catalogue.data.gov.bc.ca
Presenter
Presentation Notes
Enabler: Open Data License Enabler: CKAN license required open code, which led to …
Province of BC github
34
github.com/bcgov
Presenter
Presentation Notes
…github. This then opened up other uses, including R projects. Enabler: Apache 2.0 and Creative Commons licenses
BC Developers’ Exchange
35
bcdevexchange.org
Presenter
Presentation Notes
And having a github site then allowed the creation of the BC Developers’ Exchange. We have used the DevEx for small scale projects (remember the model rocket!) for R development … and if you’re an R developer, keep your eyes peeled for future opportunities
36
Finding like-minded people
No, not like that
Presenter
Presentation Notes
First, making a Duplicate of yourself is NOT how to find like minded people. This is a terrible way to approach finding like-minded people, as Calvin eventually discovered.
37
Finding like-minded people
Presenter
Presentation Notes
This image of Space Shuttle crew STS-116 that orbited in the International Space Station is a better image. https://en.wikipedia.org/wiki/STS-116 The connections that have been made through the open data initiatives, the DevEx, and github have been with people who are interested in orbiting the hairball. They are not always part of the R or data science community in the BC Government, but they are certainly people who want to orbit in their own way. You may have noticed that there were only 311 people listed as contributors to the github repos—this in an organization with 25,000 staff! And some of those 311 are private-sector developers. What we are looking for are people who are able to find the goals of the organization, and work towards those goals, instead of being obsessed with the processes.
Province of BC github
38
github.com/bcgov
Presenter
Presentation Notes
Enabler: Apache 2.0 and Creative Commons licenses
39
Staying in orbit
Presenter
Presentation Notes
Maintaining orbit
40
Building the team
- Diverse set of talents
- “Expand the boundaries of the team’s knowledge sphere” (Angela Bassa)
Presenter
Presentation Notes
Angela Bassa, “Managing A Data Science Team”, 2018-10-24 Harvard Business Review https://hbr.org/2018/10/managing-a-data-science-team Image: Duke Ellington and band
41
Professional development
- Make time!- Different modes
- Self-directed- Online- Classroom
Presenter
Presentation Notes
Image: Runs With Scissors, “drab is drab”, creative commons license https://www.flickr.com/photos/kenstein/103008634/
42
Collaboration
- within BC Stats
- within BC government- packages
- DevEx “code with us”
Presenter
Presentation Notes
Policies and procedures are a fact of life in any organization. The book by Gordon Mackenzie gives us a useful metaphor--The Giant Hairball--to understand them, and the further metaphor that it's possible to go into orbit around them. BC Stats, in our adoption and implementation of R, has gone into orbit. And we intend on staying there. But it's not just branches of the larger organization--like BC Stats is part of the British Columbia government--that have to deal with The Hairball. Individuals confront it too, and it can be frustrating. Here's professor who has identified that his personal style is at odds with The Hairball:�
43
New tools
- shinyapps.io- NLP and LDA
packages
Presenter
Presentation Notes
shinyapps.io The cloud gives Hairball maintainers the heebie jeebies. And this is one of those places where the policies and laws are, in some applications, entirely reasonable. In Canada, we’ve enacted privacy legislation both nationally and provincially. These, in general, say that the storage of the personal data of Canadians has to be in Canada. Using a cloud service outside Canada is legally prohibited, and in other cases adds complexity. “Once your data is transferred outside of Canada, it becomes subject to the laws of the country where the data is stored. For instance, if you send data to the U.S. for processing — or if you worked with a cloud vendor located in the U.S. — customers personal data would then be subject to U.S. law, and law enforcement agents in the U.S. could gain access to search data held by American service providers.” https://www.servercloudcanada.com/2017/10/privacy-law-canada/ There’s also questions of the intellectual property associated with that data or application. So the idea of a cloud service for, say, a data visualization, has not been in the cards. But I realized that the open data license and the open code and content licenses (associated with the github repos) opened the door for the use of a cloud service, if it we limited the data that was served up. And voila, we now have a shinyapps.io license, and we will soon be making public some applications that I mentioned earlier.
44
Supporting R development
- Package enhancement - expand functionality
- Package creation- data
45
Community
Presenter
Presentation Notes
David Smith, 2017-06-23, “The R community is one of R’s best features” https://blog.revolutionanalytics.com/2017/06/r-community.html Bcgov useR day Victoria RLadies
46
…inside the BC Public Service…
Presenter
Presentation Notes
The BC Government employee contributions to the R community - bcgov-useR events https://github.com/bcgov/bcgov-useR Community meetup groups, including Victoria R Ladies
47
useR! 2020 – Vancouver!
…and around the world!
Presenter
Presentation Notes
Developing a proposal for Vancouver to host useR! 2020
48
It’s personal, too
Presenter
Presentation Notes
But orbiting the Hairball is not just about your piece of the organization—in my case, helping BC Stats and the rest of the R community in the BC Government orbit around the Hairball. It’s also about you personally. Jeff Leek, Process versus outcome productivity https://simplystatistics.org/2018/04/19/process-versus-outcome/
49
“Find the goals of the organization that touch your heart and release your passion to follow those goals.“
- Gordon Mackenzie
Presenter
Presentation Notes
Gordon Mackenzie gives us a tool as individuals, which will help us better cope when we encounter The Hairball in our day-to-day lives. The sub-title of the book is "A Corporate Fool's Guide to Surviving with Grace". If we recognize that The Hairball exists, it can become easier--more tolerable--for us to deal with "process productivity". As Mackenzie puts it, we need to "find the goals of the organization that touch your heart and release your passion to follow those goals." [^8] > It is a delicate balance, resisting the hypnotic spell of an organization's culture and, at the same time, remaining committed from the heart to the personally relevant goals of the organization. But if you can achieve that balance and maintain it, you will be out of the Hairball and into Orbit, the only place where you can tap your one-of-a-kind magic, your genius, your limitless creativity.
50
Presenter
Presentation Notes
So now this is an image I carry in my head – that’s me! I have found goals of the organization that spark my passions, and I have found like-minded people—inside BC Stats, elsewhere in the BC Public Service, and in the R community—and together we’ve gone into orbit.
