Download - Wealthfront's Query Engine
![Page 2: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/2.jpg)
Roadmap
• Query Engine • Queries • Services • Remote Query InvocaKon
![Page 3: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/3.jpg)
The Query Engine is
• a plaMorm to build distributed services quickly
• designed with testability in mind • powering all of Wealthfront’s backend services • running on the JVM
![Page 4: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/4.jpg)
The Query Engine is not
• a web framework
• a RESTful web service – a query is a funcKon, not a resource
• Java-‐specific
![Page 5: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/5.jpg)
Queries
• First-‐class ciKzens – Queries can be passed around and later invoked
• Serializable – Queries can be persisted
• Closed with their dependencies • Composable units of work
– Queries can invoke other queries – Queries can produce other queries
• Entry points into our backend services – Services can invoke queries on other services – Queries can also be invoked from the command line
![Page 6: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/6.jpg)
Queries (concretely)
• Queries are classes • Constructors define input parameters • Instances are invokable using a driver • Dependencies can be requested at run-‐Kme
• InvocaKons produce a result • Queries are easy to test
![Page 7: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/7.jpg)
UNIX Processes
Process
stdin Environment
stdout Return code
stderr
![Page 8: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/8.jpg)
Queries
Query
Arguments Dependencies
Result ExcepKon
![Page 9: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/9.jpg)
Invoking a Query
Query Class
Driver
Query Instance
Arguments
Dependencies
Result
![Page 10: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/10.jpg)
Result
Run
Query Instance
Invoking a Query
Scoping Monitoring Retrying TransacKng InjecKng
Dependencies
![Page 11: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/11.jpg)
Services
• A collecKon of Queries – Usually with a similar purpose, e.g. all the queries related to customer management
• Able to saKsfy the dependencies required by its Queries – E.g. access to the customer database, connecKon to the NASDAQ NLS feed, …
• Queries can be installed in different services
![Page 12: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/12.jpg)
Remote Query InvocaKon
• Queries are remotely invoked by doing an HTTP POST request
• Arguments are encoded in the HTTP request
• Results are returned in the HTTP response
![Page 13: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/13.jpg)
Request SerializaKon
• Most of our services rely on the so-‐called “qp0p1” serializaKon – The simple name of the invoked query is passed with the q parameter
– The nth argument is passed with the pnth parameter
– Arguments are serialized and de-‐serialized to and from strings using converters
![Page 14: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/14.jpg)
Hello(@OpKonal(“World”) String): String
Instan&a&on Serializa&on
new Hello(“Bob”) q=Hello&p0=Bob
new Hello(null) q=Hello
![Page 15: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/15.jpg)
[julien@glados ~]$ curl um0:8085 -‐-‐data 'q=Hello&p0=Bob’ -‐v
> POST / HTTP/1.1> Content-‐Length: 14 > Content-‐Type: applicaKon/x-‐www-‐form-‐urlencoded
> > q=Hello&p0=Bob
< HTTP/1.1 200 OK < X-‐KC-‐TraceToken: 962dcf36-‐9b25-‐4731-‐a022-‐e054b925637c
< Server: kawala < Content-‐Length: 11
Hello, Bob!
![Page 16: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/16.jpg)
Request De-‐serializaKon
• Services’ request interpreters de-‐serialize the HTTP POST requests and create instances of the query class using the specified arguments
• Then, they invoke the query instance using a driver
![Page 17: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/17.jpg)
Request De-‐serializaKon
Query Class
Request Interpreter
Query Instance
Arguments
HTTP POST request
Driver
![Page 18: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/18.jpg)
Smart Clients
• Queries can be remotely invoked from Java code using smart clients
• Smart clients analyze the bytecode to recover the arguments passed to the query constructor and generate the HTTP POST request
• Smart clients also de-‐serialize the result from the HTTP response (see “Result SerializaKon”)
![Page 19: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/19.jpg)
Smart Clients
Query Class
Smart Client
Query Instance
Arguments
HTTP POST request
![Page 20: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/20.jpg)
The Life of a Remote Request
Query Class
Smart Client
Query Instance
Arguments
HTTP POST request
Query Class
Request Interpreter
Query Instance
Arguments
Driver
Client Server
![Page 21: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/21.jpg)
Constructor Requirements
• In order to be analyzable, query constructors must follow strict requirements – Basically, they should only assign their arguments to fields
hwps://github.com/wealthfront/kawala/wiki/InstanKators
![Page 22: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/22.jpg)
Sugar for the Constructor
• @OpKonal(“2011-‐02-‐25”) LocalDate • @OpKonal(“false”) boolean
• OpKon<LocalDate> • @PosiKve int age
![Page 23: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/23.jpg)
Early ValidaKon
• Arguments are validated when instanKaKng a serialized query thanks to the converters
[julien@glados ~]$ ikq um0 GetUser foo
HTTP Error 400: For input string: "foo” X-‐KC-‐TraceToken: 962dcf36-‐9b25-‐4731-‐a022-‐e054b925637c
Server: kawala ConnecKon: close
![Page 24: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/24.jpg)
Result SerializaKon
• Results are usually serialized as – JSON – Protobuf
• The service has the responsibility to serialize the result of a query with the proper serializaKon method
• Smart clients rely on the same logic to select the proper de-‐serializaKon method
![Page 25: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/25.jpg)
Result SerializaKon
Return Type Addi&onal Constraints Serializa&on
ConverKble type* JSON value**
T T annotated with @EnKty JSON object
List<T> T annotated with @EnKty JSON array
Set<T> T annotated with @EnKty JSON array
T extends Message Protobuf
List<T extends Message> Protobuf array
* As specified in KachingMarshallers ** JSON strings are returned without surrounding quotes
![Page 26: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/26.jpg)
The Life of a Remote InvocaKon
Query Class
Smart Client
Query Instance
Arguments Query Class
Request Interpreter
Query Instance
Arguments
Driver
Client Server
Result
Serializer
Result
![Page 27: Wealthfront's Query Engine](https://reader031.vdocuments.net/reader031/viewer/2022020217/5562566bd8b42a1b4b8b5228/html5/thumbnails/27.jpg)
Query InvocaKon Summary
• Local InvocaKon QueryExecutor executor = …
User user = executor.submit(new GetUser(Id.<User> of(10));
• Remote InvocaKon SmartClient<UM> um = … User user = um.invoke(new GetUser(Id.<User> of(10));