how and why we evolved a legacy java web application to scala... and we are still alive!
TRANSCRIPT
How and why we evolved a legacy java application to scala
And we are still alive !24/06/2015
I am
@karesti
The Software Dream
Software is more like Madonna
FACTS
Web applications get old (very) fast
Continuous small refactoring does not avoid long-term technical debt
From strach Refactoring
Continuous Dilemma
2014
French Job Search Website Launched in 2000
2008
Problems in 2014
High Cost Adding New FunctionalitiesHigh Technical Debt
Coupled Architecture
S O A Spaghettis Oriented Architecture
Lack of real KPI
In a 100% Linux Environment
Monolithic Architecture
DAO
ServiceBatch
MVC
RDMS
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
Connected User BoardCV, Mail Alert, Newsletter
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
Data
Mailing App
Connected User BoardCV, Mail Alert, Newsletter
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
Data
Mailing App
External App
Connected User BoardCV, Mail Alert, Newsletter
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
Data
Mailing App
External App
Mobile Version
Connected User BoardCV, Mail Alert, Newsletter
DAO
ServiceBatch
MVC
RDMS
Anonymous UserJob Search, Detail, Newsletter
Jobs
Jobs
External App
Data
Mailing App
External App
Mobile Version
Connected User BoardCV, Mail Alert, Newsletter
DAO
ServiceBatch
MVC
RDMSExternal
App
Connected User BoardCV, Mail Alert, Newsletter
Anonymous UserJob Search, Detail, Newsletter
Non connectedAlerts, Newsletter
Jobs
Jobs
External App
Data
Mailing App
Mobile Version
Partners
How do we fix this
Target
Where do we start
DAO
ServiceBatch
MVC
RDMSExternal
App
Jobs
Jobs
External App
Data
Mailing App
Mobile Version
Partners
Connected User BoardCV, Mail Alert, Newsletter
Anonymous UserJob Search, Detail, Newsletter
Non connectedAlerts, Newsletter
Mars – September 2014
User Board
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
User BoardUser Board Front-End
REST API
Mongo
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
External App
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
External App
DAO
ServiceBatch
MVC
RDMS
Data
Mailing App
Where do we start ?User Board Front-End
REST API
Mongo
Partners
External App
Data
Batch
API First
Macro
Operation
Focus on technical choices
Main Language
• Growing and Solid Community
• Powerful Frameworks and Utilities
• JVM
Backend
• REST oriented• Template Type Safe• Hot Reloading• “Simplifies” Scala• Reactive programming
Batch System
• Actors
• Scale Up – concurrency
• Scale Out – remoting
• Fault Tolerance
Front End
• Sass built in with play
• AngularJS, popular, community, experience
• Foundation, solid and easy CSS framework
Database
• Flexible schema• Document oriented makes sense with CVs• Low transactions• Application code rules the database schema– Already the case with SQL Server
• No DBA*– We rule the database as dev
Migrating Data Challenge
Status
• +10 years of candidate data
• Crucial for business
• Cannot fail really, C A N O T F A I L !!
Strategy
• Start migration as soon as possible– Started in April – Mai
• Migrate data incrementally
• Verify as much as possible
• Legacy ID
Akka Actors
• One actor per data
• Concurrence execution when possible
• Handle updates for the crucial moment => between SQL Server Stop and MongoDB Up
Madrid MUG
This section is specially dedicated to the Madrid MUG Members
SQL Model
• 8 tables for the user profile• Complex joins• SQLServer => Lost in a Linux World• Devs => Backup, dump …
(Very) Simplified Schema
MongoDB Model
• 1 document / profile• Object <> Document• Simplified model– Option Scala
• Using Reactive Mongo Driver• Using Jongo in Java
Mongo Collections
Model choices
• ++ reads / -- writes• Stable Reference data *– Sometimes sectors can change…
• Low and non risky transactions– Ex : user deletes account
Testing
Unit Testing
• Using Specs framework
• Using Mockito – Not as useful as in Java
• Some tests are not necessary – Constructors, Builders … Scala Type Safe and
Immutability
Testing the Front END
• Unit testing JS with Karma
• Selenium – Very fragile tests– Proxy Nightmare– Endless navigator problems– Just vital tests after production
API Tests are CRUCIAL
Testing the REST API
• No Mocking MongoDB
• Using Embedded Mongo
• Start and Stop Mongo once for every test– DRY data is a hard part
https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo
Always thinking on KPI
Build Measure Learn
SCALA USER GROUP
This section is specially dedicated to the Madrid Scala User Group.Thank you to Nouhoum Traoré who spoke about it at scala.io in Paris
Front-end
● Client API
● No DB Access
● Mostly Javascript Code
Front-end
● 9 % Scala● 26.2 % CSS
●64.8 % JS
Le frontend : asset pipeline
pipelineStages := Seq(rjs, digest, gzip)
Le frontend : asset pipeline
curl http://keljob.com/assets/js/e454f1013e30b783818c8efaf3a8e3a5-startup.js
HTTP/1.1 200 OKCache-Control: public, max-age=31536000Content-Length: 171997Content-Type: application/javascript; charset=utf-8Date: Tue, 14 Oct 2014 22:39:23 GMTETag: e454f1013e30b783818c8efaf3a8e3a5Last-Modified: Wed, 08 Oct 2014 12:35:10 GMT
Compressing content
import play.api.mvc._import play.filters.gzip.GzipFilter
object Global extends WithFilters(new GzipFilter()) {
...}
API : links on JSON
implicit val AccountWrites = new Writes[Account] {
override def writes(account: Account): JsValue =
Json.obj(
"id" -> account.id,
"email" -> account.email,
"creationDate" -> account.creationDate.toString(),
"links" -> Json.obj(
"self" -> routes.AccountDetailCtrl.get(account.id).url,
"alerts" ->
routes.JobAlertDetailCtrl.getAlertsOf(account.id).url,
"cv" -> routes.CvDetailController.getCvOf(account.id).url
)
)
}
Single responsibility Object : Controllers@Singleton
class AccountValidationController @Inject() (
accountValidator: AccountValidator) extends Controller
{
def validateAccount(token: String) = Action.async
{ request =>
accountValidator.validate(token).map {
case Some(account) => Ok(Json.toJson(account))
case _ =>
UnprocessableEntity(Json.toJson(InvalidToken))
}
}
Single responsibility Object : Servicesclass AccountAuthenticator @Inject() (...)
class AccountValidator @Inject() (...)
class AccountCreator @Inject() (...)
class AccountSettingsUpdater @Inject() (...)
Single Responsibility Object : Actors
import akka.actor._
...
class CvExporter(...) extends Actor { def commonBehavior(): Receive = ??? def deleteBehavior(): Receive = ??? def updateBehavior(): Receive = ???
def receive = commonBehavior orElse updateBehavior orElse deleteBehavior
}
Error Handling in Services
sealed trait NewsletterError
case object InvalidNewsletterActivationToken extends
NewsletterError
case object NewsletterUpdateError extends NewsletterError
class NewsletterActivator @Inject() (...) {
def activate(code: String): Future[Either[NewsletterError, Boolean]]
= ???
}
Error handling in controllers
class NewsletterActivationController (
newsletterActivor: NewsletterActivator) extends Controller {
def activate(token: String) = Action.async { request =>
newsletterActivor.activate(token).map {
…
case InvalidNewsletterActivationToken => ???
case NewsletterUpdateError => ???
…
}.recoverApiError("Oops !!!")
}
}
Error handling in controllers
implicit class ApiErrorRecover(result: Future[Result]) {
def recoverApiError(message: String) =
result recover {
case NonFatal(e) => InternalServerError(
Json.toJson(SimpleError(message))
)
}
}
The Team
The (original) Team
• 3 Developers and a Product Owner– Legacy + Backend + Scala– Full-Stack– Java + Backend + MongoDB
• From 5-10 years of experience• People who are able to leave their confortable
coding zone• Want to communicate
Project Management Method
SCRUM
KANBAN
Programing MF
(SOME) DIFFICULTIES
DEFINING THE INITIAL SCOPE
MVP
MVP
Maximal Viable Product
DEALING WITH NO TECHNICAL PEOPLE
Demo
We have a situation …
Y
FIF Pattern
Fancy Interface First
SCALA
Warning ! This section might contain some trolls
Personal journey to Scala
SIMPLE ??? Build Tool
"de.flapdoodle.embed" % "de.flapdoodle.embed.mongo" % "1.46.0” "org.mongodb" %% "casbah" % "2.5.0"
Reactive Futures …
Implicits
accountFuture.filter(_.isDefined).map(_.get)
Cake Pattern
DI Framework vs Cake Pattern
@Singletonclass LoginController @Inject() (accountAuthenticator: AccountAuthenticator) extends Controller
trait LoginController { this: Controller with UserServiceComponent =>
object LoginController extends LoginController with Controller with MongoDbUserServiceComponent
Loving Case Class And Constructors
case class Account( id: Option[BSONObjectID] = None,
email: String, creationDate: DateTime,
optin: Boolean = false, optoutScenario: Option[DateTime] = None, source: String = Account.DEFAULT_SOURCE,
deletionDate: Option[DateTime] = None)
Account(“[email protected]”, creationDate = creationDate,
optin = true)
After 2-3 months as happy as being at Machu Picchu
Loving both …
September – December 2014
Once in production
– Creating accounts more easily– Parsing CV on upload– Can apply with the CV – Follow applications
• And other awesome stuff built fast and furiously !
January – March 2015
Search and Relooking
Relooking and positioning
BatchDataMailing
App
2015Web Front-
End
User REST API
Mongo
Partners in WIP
External App Data
Search REST API
Elastic Search
Batch
SEOMost Important Challenge
Challenge 2 : Performance
• Gatling – Play!
• Comparing performance