Solving a business problem in just two weeksusing machine learning

Lucas F Navarro and Pedro Suguimotolucas.navarro@getninjas.com.brpedro.suguimoto@getninjas.com.br

PAPIs Connect - São Paulo 2017

GetNinjas’ Data Science Squad

Product OwnerBs’ in Computer Science at UFSCarMs’ in Artificial Intelligence at UFSCar

Data ScientistMechatronics Engineer from

USP/CAASO

Data ScientistBs’ in Applied Maths at UNICAMPPhD’ in Artificial Intelligence at USP/POLI

Python DeveloperBs’ in Information System at USP/EACH

BackEnd DeveloperComputer Engineer from

USP/POLI

Agenda

1. How does GetNinjas Work

2. The current Business Model and why we chose it

3. The big problem that has emerged

4. How we solved this problem in just two weeks

5. Results we achieved

6. Touchpoints

How does GetNinjas work?(Client)

1. Choose what kind of service that you need.

2. We indicate up to 4 pros at your region

3. Hire a Pro to solve your problem

How does GetNinjas work?(Pro)

1. Choose a service request from the store

2. Use credits to unlock the contact of the client

3. Get in touch with the client to be hired

OVERVIEW

2,000,000services hired in the last year

200,000professionals registered

R$ 210M+in transactions concluded (GMV) over the last 12 months

100+categories ranging from electrician to english teacher to lawyer and so on...

We were performing a staged rollout of a new pay-per-lead model, with significant improvements to the pros' experience

Old ModelSubscription

- Pro Pays Fixed Value Per Month- Receives up to

30 contacts

New ModelBundle

- Pro Purchases a Bundle of credits- Use credits to

purchase contacts

Vs

First Launch: South (Sep/18)

Second Launch: North + Center (Oct/23)

At last:SP & RJ (2017)

Main improvements:

● Pros can purchase how many contacts they want, when they want

● Clients will always receive interested Pros

● More feasible to balance Supply & Demand

Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered

Next launch was in two weeks!!!

29%of the requests were not

being sold

We would have 43.500 clients/month without response

(Around 0.5M per year)

We made quantitative and qualitative analyses to figure out what causes this rate

Some of the possible reasons:

○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits

How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.

Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered

Next launch was in two weeks!!!

29%of the requests were not

being sold

We would have 43.500 clients/month without response

(Around 0.5M per year)

We made quantitative and qualitative analyses to figure out what causes this rate

Some of the possible reasons:

○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits

How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.

Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered

Next launch was in two weeks!!!

29%of the requests were not

being sold

We would have 43.500 clients/month without response

(Around 0.5M per year)

We made quantitative and qualitative analyses to figure out what causes this rate

Some of the possible reasons:

○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits

How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.

The problem gets bigger!

Each point is a “Mini GetNinjas”, with 100 categories

300 cities

100 categoriesx

30k pairswhich we have to deal with that problem

What if we could predict that a client request wouldn’t be purchased?

1st option, tell the client the truth

“Unfortunately we don’t have Pros available to solve your problem at this moment”

2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔

“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”

11%of the Free Pros were purchasing a bundle after seeing the services store at our App

Only 4% of our subscribed pros were payees at that time

We had the potential to more than double our payees base if we sustained that conversion rate

What if we could predict that a client request wouldn’t be purchased?

1st option, tell the client the truth

“Unfortunately we don’t have Pros available to solve your problem at this moment”

2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔

“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”

11%of the Free Pros were purchasing a bundle after seeing the services store at our App

Only 4% of our subscribed pros were payees at that time

We had the potential to more than double our payees base if we sustained that conversion rate

What if we could predict that a client request wouldn’t be purchased?

1st option, tell the client the truth

“Unfortunately we don’t have Pros available to solve your problem at this moment”

2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔

“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”

11%of the Free Pros were purchasing a bundle after seeing the services store at our App

Only 4% of our subscribed pros were payees at that time

We had the potential to more than double our payees base if we sustained that conversion rate

How could we predict that a client request wouldn’t be purchased?

We had already figured out the reasons that a service was not sold, so we used it as features and started to test some machine learning models using R.

Main R packages used:

For data prep:○ tidyr○ dplyr

ML algorithms:○ monmlp (neural net)○ e1071 (naivebayes)○ party (decision trees)

Utilities:○ ROCR (ROC curve)○ SDMTools (confusion matrix)○ Boruta (feature selection)○ Pool + RPostgreSQL (integration

for our data warehouse - RedShift)

Some of the features:

● Number of Pros with credits*● Avg Distance from pros*● Number of highly reviewed pros*● Price● Does service has a text description?

Class: Was the request sold? (boolean)

We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud

*Just pros inside the request area (inside a radius)

From R to Python, Fujin was Born!

We came up with a Neural Network with great performance:

○ With this specificity rate, just 30% of the unsold requests clients would remain without an answer

○ With this sensitivity rate, we would distribute only 10% of the good requests for free

After we validated the model, we implemented it in python to go live:

○ We used the basic DS set (scikit-learn, pandas, numpy)○ Alse we used Neurolab for the MLP algorithm○ We used Flask to create the prediction microservice

70%of specificity

First Model

79%of accuracy

*Python is the main Data Science language at GetNinjas

90%of sensitivity

We called the project Fujin!

How fast this entire project was delivered?

To have a functional ML model to predict unpurchased requests we spent just around 2 weeks!

*The first distribution algorithm was a random distribution to Free Pros

In Its first month live, Fujin were able to predict 68% of the unpurchased requests

Unpurchased

Predicted/Distributed

Contacted

Hired

41%

28%

9%

5%

Service

RequestsWith north+center regions, unpurchased request rate grown from 29% to 41% due sparsity of S&D in this new regions

With Fujin online, the number of unanswered clients were just 13% (instead of 41%)

Fujin also helps to increase our Gross Merchandise Volume(GMV) through hiring rate

*Numbers from our requests of November/2016

Fujin was also helping to activate Free Pros into Payees

*Numbers from our requests of November/2016

≈R$25k in purchases from Pros after they received a free client contact from Fujin

65% of this pros didn’t even had our App installed

⚠ Only 14% of the distributed leads were read by the Pros

Already had our App installed Downloaded App post Fujin lead Didn`t Download

Towards to the Future

● The distribution algorithm○ Pick the best pros to send free requests○ Determine the amount of free pros to send the contact from a request given the Fujin’s

feedback

● Enhance the model with more features○ We have more service categories with distinct characteristics nowadays○ We have SP and RJ state (higher density regions than before)

● Features to increase the visualization rate of free distributed requests○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros

Touchpoints

● The non-bureaucratic/non-hierarchical organization of a StartUp company was crucial for us to develop Fujin in such short time

● ML is amazing, in just two weeks we had a running model to solve the biggest part of our problem

● ML is super cool, but most of the times we solve our problems with much simpler approaches

⚠ Do not create a super complicated solution and later adjust your problem to it, start from the problem, and climb to the complex solution as you need it.

Thanks for your attention!!!

Questions?