managing large (and small) r based solutions with r suite

Copyright (c) WLOG Solutions

Managing large (and small) R based solutions with

R SuiteWit Jakuczun,

WLOG Solutions2017-09-29

http://wlogsolutions.com

http://rsuite.io

Copyright (c) WLOG Solutions 2

Open-source is a leader the race in advanced analytics

using GPU engines and power of its community.


Example?


4000x4 elastic-net models (CV-5) for 45Kx10K datasetin 1,5 minute!

http://www.youtube.com/watch?v=LrC3mBNG7WU


Join 21st centuRy today!


What is R?

7


Dynamically interpreted general programming language


Stable open-source productdeveloped by R Foundation

since ~1995 year.


Created for data analysis.


flights %>%

group_by(year, month, day) %>%

select(arr_delay, dep_delay) %>%

summarise(

arr = mean(arr_delay, na.rm = TRUE),

dep = mean(dep_delay, na.rm = TRUE)

) %>%

filter(arr > 30 | dep > 30)

z <- scaled_input %>%

layer_convolution2D(c(5,5), 32, pad = TRUE) %>%

layer_max_pooling(c(3,3), c(2,2)) %>%

layer_convolution2D(c(3,3), 48) %>%

layer_max_pooling(c(3,3), c(2,2)) %>%

layer_convolution2D(c(3,3), 64) %>%

layer_dense(96) %>%

layer_dropout(0.5) %>%

layer_dense(num_output_classes,

activation = activation_softmax())

Data preparation Predictive model building


R is a community.


CRAN10K+ packages

GitHubmore and more

popular


However, out-of-the-box R does notprovide many essential features requiredIn a large-scale production deployment!


R Software DevelopmentWhat is a large scale?


R software development vs

R scripting


Large scale ~ 10K+ LOCSmall scale ~ 1K LOC


Examples of large scale projects


Historicaldata

Traffic forecasting

Workforce optimiser

Busines rulesHuman intervention

ResultsEfficiency

curves

Workforce optimisation


Historicaldata

Cash flow “forecasting”

Optimisation phase I

Optimisation phase II

Business rulesHuman intervention

Results

FONG

Cash optimization


Production DevContinuousintegration

Version control

R Studio Server

Data-science team

http://rsuite.io

http://rsuite.io


R Software DevelopmentWhat does R give us “out-of-the-box”?


Dev

Version controlContinuous

Integration & Deployment

Prod

Software development process


What is nice about R?


Package help system

Package dependency

system

External data in packages

Vignettes Tests


Where does R have its rough edges?


CRAN (MRAN) Github Other

Dev environment

Installed packages

Local CRANSource code repo


install.packages(“ggplot2”)


CRAN (MRAN) Github Other

Dev environment

Installed packages

Local CRANSource code repo


TeaserAlcohol concentration in blood

Copyright (c) WLOG SolutionsCopyright (c) WLOG Solutions 31

Instruction (Windows only!)

1. Install vanilla R 3.4.22. Download package

(https://goo.gl/RnZAQg)3. Unzip4. Open CMD5. Rscript R/master.R \

--port=71376. In browser open

http://localhost:7137

http://www.sumsar.net/blog/2014/07/estimate-your-bac-using-drinkr/


https://goo.gl/RnZAQg

http://localhost:7137



R Software DevelopmentR Suite walk-through


Preliminaries


Install R

https://cran.r-project.org/bin/windows/base/R-3.4.2-win.exe

https://cran.r-project.org/bin/windows/base/R-3.4.2-win.exe


Install R Suite

35

http://rsuite.io/RSuite_Download.php




Development cycle with R Suite

36

1. Start a project2. Add a package to the project

a. Develop the package (using devtools)3. Add dependencies to the project4. Build the package5. Build a deployment package


Open tutorial

http://rsuite.io/RSuite_Tutorial.php?article=basic_workflow.md

http://rsuite.io/RSuite_Tutorial.php?article=basic_workflow.md


Project structure


deployment

logs

packages

R

tests

import

export

work

Where all the packages are installed.


deployment

logs

packages

R

tests

import

export

work

Where all the logs are stored.


deployment

logs

packages

R

tests

import

export

work

Where all the project packages are stored.


deployment

logs

packages

R

tests

import

export

work

Where master scripts (~ main) are stored.


deployment

logs

packages

R

tests

import

export

work

Where project tests are stored.


deployment

logs

packages

R

tests

import

export

work

(Optional)

Raw data to be imported.


deployment

logs

packages

R

tests

import

export

work

(Optional)

Results generated by our R program.


deployment

logs

packages

R

tests

import

export

work

(Optional)

Temporary results.


Configuration files.


RSuiteVersion: 0.10.212RVersion: 3.3Project: myprojectRepositories: MRAN[2017-09-28], Dir[]Artifacts: config_templ.txt

Example

PARAMETERS● RVersion - R version for this

project● Project - project name● Repositories

○ MRAN[YYYY-MM-DD],○ S3[URL]○ URL[URL]○ Dir[]

● Artifacts - what else should be added to deployment package.


LogLevel: INFON_days: 365solver_max_iterations: 10solver_opt_horizon: 8

Example

config_templ.txt (config.txt)

● LogLevel - level for loggers● _templ.txt - template for config● config.txt - deployment version


master.R

Here goes your code...

Autodected path of master.R

There can be many master scripts!


Put all logic into packages


Select external packages carefully.And control their versions!


data.table


print is not for logging.

Forbidden


loginfo("Phase 1 passed")

logdebug("Iter %d done", i)

logwarning("Are you sure?")

logerror("I failed :(")


pkg_loginfo("Phase 1 passed")

pkg_logdebug("Iter %d done", i)

pkg_logwarning("Are you sure?")

pkg_logerror("I failed :(")


Automate building, deploying, testing, etc.


Wit Jakuczun, PhD

[email protected]

62

Field tested R ecosystem for Enterprise

http://rsuite.io

mailto:[email protected]

http://rsuite.io

http://rsuite.io

managing large (and small) r based solutions with r suite

Data & Analytics