building a web application with ontinuation monads
TRANSCRIPT
1
Building a web application withcontinuation monads
Seitaro Yuki / @pab_tech / DWANGO Co., Ltd.
2
OutlineIntroduction of DWANGO and Niconico
The account system of Niconico and its tasks
Problems of component technologies of existing webframeworks
How to construct web applications using continuationmonads
Updating the component using indexed continuationmonads.
3
Introduction of DWANGO and Niconico
DWANGO is a company that operates a video sharing andlive broadcast platform Niconico.
Niconico has 50 million accounts and 2.5 million ofpremium members who pay 500 yen a month.
It has been known as one of most major streaming mediaservice in Japan.
4
Functional programming in DWANGO
DWANGO also has been known as a company that hasadopted functional programming languages in Japan.
DWANGO is using a variety of functional languages such asScala and Erlang.
DWANGO publishes Japanese Scala textbook for in-housetraining to Github. https://github.com/dwango/scala_text
5
Decomposing Niconico services
Previously Niconico was the one large PHP system that hasmillions of lines.
We are separating the common foundation system fromthe large PHP system using Scala
And separating the content delivery system using Erlang
As a result, it became easy to add functions to the system,and resistant to failure.
6
The account system of Niconico
DWANGO has a variety of services such as e-books andslide services other than video and live streaming.
The functionality related to users has been aggregatedinto the account system.
The tasks of the account system are the following.
User registration
User authentication
Operation user information
Decision whether premium or not
7
Various interfaces required for the account system
The account system is used for various services, and theyhave many devices.
Therefore the account system is required variousinterfaces.
8
User registrationRegistration on a registration page
Registration via an API
Registration with premium registration
Registration with connectivity verification of E-mail or SMS
9
User authenticationAuthentication from E-mail address and password
Authentication using OAuth such as Facebook and Twitter
2-step authentication with TOTP
2-step authentication with E-mail
10
Various responsesHTML、JSON、XML
Redirects
Japanese, English and Chinese
11
Technical requirementsCSRF check using a token
Session management using HTTP Cookies
Adding user tracking ID to the response
Adding CORS header to the response
12
Error handlingIn Web applications, it should return a correct formatresponse as much as possible, even if a critical erroroccurs.
13
Components of web applicationsThe various factors described so far will be used incombination.
So, we want a component technology to decompose a webapplication.
And, we want to assemble freely components to constructa web application.
14
Typical component technologies of web applications.
public interface Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain);}
doFilter in Java Servlet Filter
ServletRequest means an HTTP Request
ServletResponse means an HTTP Response
FilterChain means a next Filter
15
Filter Examplepublic class ExampleFilter implements Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain) { // preprocess for request chain.doFilter(request, response); // postprocess for response }}
You can put a preprocess before calling the next chain.
You can also put a postprocess a�er calling the next chain.
These nested structures can be seen in various languagessuch as WSGI of Python, Rack of Ruby.
16
Typical Web application sequence diagramWeb application
Web Browser Web Server Filter A Filter B Main Processing
request
request
request
request
response
response
response
response
17
Example: Authentication verificationIt assumes that a user logins elsewhere and there is asession in HTTP cookies before this operation.
This component stores the session information in serverstorage such as Redis.
The component compares the cookie information and theserver information to check whether authenticated.
If the session is correct, then this component passes thesession information to the next one.
Otherwise, it redirects to a login form.
18
Authentication verification sequence diagram
Web Browser Web Server Authentication Filter Main Processing
request
request
alt ["Authentication is successful"]
request
request
response
response
response
["Authentication is failed"]
redirect
response
19
Example: CompressionA compression component doesn't do anything to arequest.
This component will compress a response only if therequest has an Accept-Encoding.
20
Compression sequence diagramWeb Browser Web Server Compression Filter Main Processing
request
request
request
request
response
response
alt [There is "Accept‒Encoding: gzip"]
response compress response
[There is not "Accept‒Encoding: gzip"]
response do nothing to response
response
21
Problems of the component technologies ofexisting web applications
public interface Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain);}
This filter method doesn't have much information on thetype and parameters.
We don't know what kind of parameter is passed to thenext component, what combination of components canbe.
22
The component of Play framework in Scalatrait ActionFunction[-R[_], +P[_]] { def invokeBlock[A](request: R[A], block: P[A] => Future[Result]): Future}
The type parameter R represents a type of a Request.
The type parameter P represents a type of a transformedRequest by this function.
The type parameter A represents a type of a body of arequest.
ActionFunction represents a transformation from R to P.
23
Example: AuthenticatedFunctionclass AuthenticatedRequest[A](session: Session, request: Request[A])
object AuthenticatedFunction extends ActionFunction[Request, AuthenticatedRequest] {
def invokeBlock[A]( request: Request[A], block: Authenticated[A] => Future[Result]): Future[Result] = ???}
A component to authenticate and take a sessioninformation from the request.
AuthenticatedRequest is added an information of asession to the Request.
AuthenticatedFunction is a function that converts Requestto AuthenticatedRequest.
24
Problem of ActionFunctionclass LanguageRequest[A](language: Language, request: Request[A])
object LanguageFunction extends ActionFunction[Request, LanguageRequest
AuthenticatedFunction.andThen(LanguageFunction) // Can not!
LanguageFunction takes a Language parameter from theAccept-Language of Request.
This function can't be combined withAuthenticatedFunction for the mismatched types of therequest.
25
Introduction of components usingcontinuation monads
26
ContinuationA
Continuation
A R
The continuation represents the rest of computation at agiven point in execution.
A is some point of execution.
R is the result of the whole computation.
That means A to R is the continuation at A.
27
Continuation-passing style (CPS)Caller A B R
pass a continuation after A
altnot execute a continuation
pass a continuation after B
altnot execute a continuation
pass a continuation after R
CPS is a programming style that continuations are first-class.
28
Continuation monadcase class Cont[R, A](run: (A => R) => R) { def map[B](f: A => B): Cont[R, B] = Cont(k => run(a => k(f(a)))) def flatMap[B](f: A => Cont[R, B]): Cont[R, B] = Cont(k => run(a => f(a).run(k)))}
A type parameter A means a value in the given point ofexecution.
A type parameter R means a result of this monad.
A function of A to R means the continuation at A.
Thus a continuation monad converts a function thatreceives the continuation to a monad.
29
How to construct Web applications withcontinuation monads.
30
Authentication verificationWeb Browser Web Server Authentication Filter Main Processing
request
request
alt ["Authentication is successful"]
request
request
response
response
response
["Authentication is failed"]
redirect
response
31
Code of Authentication verification// takes a session id from a cookie from the request.def readSessionIdFromCookie(request: Request): Option[SessionId] = ???
// takes a session information from Redis using the session id.def readSessionFromRedis(sessionId: SessionId): Option[Session] = ???
// returns a response that redirects to a login form.def redirectLoginForm: Response = ???
// k is a continuation has a type of Session to Response.// verifyAuth is the authentication verification component.def verifyAuth(request: Request): Cont[Response, Session] = Cont((k: Session => Response) => readSessionIdFromCookie(request) .flatMap(readSessionFromRedis) .map(k) .getOrElse(redirectLoginForm))
32
CompressionWeb Browser Web Server Compression Filter Main Processing
request
request
request
request
response
response
alt [There is "Accept‒Encoding: gzip"]
response compress response
[There is not "Accept‒Encoding: gzip"]
response do nothing to response
response
33
Code of Compression// takes a value of the Accept-Encoding in the request.def readAcceptEncoding(request: Request): List[Encoding] = ???
// compresses the response.def gzip(response: Response): Response = ???
// compress is the Compression component.def compress(request: Request): Cont[Response, Unit] = Cont((k: Unit => Response) => { val response = k(()) // if the Accept-Encoding header contains "gzip" if (readAcceptEncoding(request).contains(Encoding.GZIP)) gzip(response) // compresses the response else response // doesn't do anything to the response })
34
Compose continuation monads using for-expression in Scala
// a core part of web application.def mainProcessing(request: Request, session: Session): Cont[Response,
def webApplication(request: Request): Cont[Response, Response] = for { session <- verifyAuth(request) _ <- compress(request) response <- mainProcessing(request, session) } yield response
The for-expression in Scala is used for composing monads.
We can compose verifyAuth, compress, andmainProcessing using for-expression.
35
Combine continuation monad transformerwith Scala Future
import scalaz._import scalaz.std.scalaFuture._
object ActionCont { type ActionCont[R, A] = ContT[Future, R, A]
def recover[R, A] (cont: ActionCont[A, A]) (pf: PartialFunction[Throwable, Future[A]]): ActionCont[R, A] = ContT(cont.run_.recoverWith(pf).flatMap)}
Scala Future is a monad has two functionalities that are
asynchronous computation and
variant type of error value.
36
Problem of continuation monad component
Components of continuation monads are very useful.
But there is an issue that a type of response is fixed.
37
Problem of a fixed type of response// takes a value of the Accept in the request.def readAccept(request: Request): List[MediaType] = ???
def toJson(entity: Entity): Response = ???
def toXml(entity: Entity): Response = ???
def toJsonOrXml(request: Request): Cont[Response, Unit] = Cont((k: Unit => Response) => { val response = k(()) // if the Accept header contains "application/json" if (readAcceptHeader(request).contains(MediaType.JSON)) toJson(response) // TypeError: Entity expected but actually Response else toXml(response) // TypeError: Entity expected but actually Response })
38
Updating components using indexed monad
So we would like to update our components with indexedcontinuation monad.
Indexed monad is a monad has an another type parametercan be changed.
We can deal with changes in a parameter of the monadusing the indexed monad.
For example, the indexed state monads can change a typeof state.
39
Indexed continuation monadO
Continuation
A R
Added type parameter O means a result type ofcontinuation.
40
Code of indexed continuation monadcase class IndexedCont[R, O, A](run: (A => O) => R) { def map[B](f: A => B): IndexedCont[R, O, B] = IndexedCont(k => run(a => k(f(a)))) def flatMap[E, B](f: A => IndexedCont[O, E, B]): IndexedCont[R, E, B IndexedCont(k => run(a => f(a).run(k))) def contramap[I](f: I => O): IndexedCont[R, I, A] = IndexedCont(k => run(a => f(k(a))))}
By distinguishing a result type of a continuation from aresult type of the whole,
We can deal with changes in a result type.
And we think that a contramap function to be a pair with amap function.
41
Composition of indexed continuation monad
Using flatMap of IndexedCont, we can composeIndexedCont[R, O, A] and IndexedCont[O, E, B] toIndexedCont[R, E, B]
E RB
OA R
OEBIndexedCont[R, O, A]
IndexedCont[O, E, B]
IndexedCont[R, E, B]
42
Example of a component of indexedcontinuation monad
def toJsonOrXml(request: Request): IndexedCont[Response, Entity, Unit] = IndexedCont((k: Unit => Entity) => { val entity = k(()) // intermediate result if (readAcceptHeader(request).contains(MediaType.JSON)) toJson(entity) // not a compile error else toXml(entity) // not a compile error })
Unlike the previous example, we can deal with entitieshave a type of an intermediate result.
43
Example of contramap of indexedcontinuation monad
def toJsonOrXml(request: Request): IndexedCont[Response, Entity, Unit] = IndexedCont((_: Unit => Response)(())).contramap { if (readAcceptHeader(request).contains(MediaType.JSON)) toJson else toXml }
We can also use contramap in the conversion process ofresponse.
44
Continuation monads in the account systemof Niconico
Actually, there are about 50 independent components inthe account system of Niconico
Let's see some of them that classified by purpose.
45
General purpose componentsComponent Purpose
FormRequestCont To get Form parameters of the request
JsonRequestCont To get Form parameters of the request
HandlerCont To wrap a domain service function
UserTrackIdCont To add user tracking id to the response
46
Components for APIComponent Purpose
AuthorityCont To verify that clients has theauthority to API
MaintenanceCont To return 503 if the service is undermaintenance
ExceptionRendererCont To convert errors to the responseof API
ResponseFormatCont To determine whether JSON orXML that the service should return
47
Components for Web pagesComponent Description
RedirectVerificationCont To verify that a given RedirectURL is contained in the whitelist
RedirectLoginFormCont To redirect the login form ifauthentication verification failed
PasswordVerificationCont To ask a password to user beforedisplaying a page
CsrfTokenVerificationCont To check a CSRF token
48
Components construct other componentsComponent Description
FlowCont To construct components such as pre-processing and error handling component
ApiCont To construct components required for API
WebPageCont To construct components required for Webpages
49
Conclusion
We have seen the construction Web application usingcontinuation monads.
Components of continuation monads is flexible tocompose and easy to understand.
We are updating our components to indexed continuationmonads to change the result type of continuation.
We can make a variety of interfaces by combining thecontinuation monad, then we can concentrate on moreimportant domain logic.