introduction to courier
TRANSCRIPT
CourierJoe Betz @ Coursera
Courier is a code generator
{"name": "Fortune","namespace": "org.example","type": "record","fields": [{
"name": "message", "type": "string"
}]
}
{ "message": "Today is your lucky day!" }
case class Fortune(message: String)
JSON Data
ScalaPegasus Schema generate
serialize / deserialize
● Extension of Apache Avro’s schema language built at Linkedin.
● Designed for natural looking JSON.
● Rich type system maps well between JSON and type-safe
languages like Scala.
● Schema language is machine readable and easy to extend.
● Tooling and language support.
Pegasus Schema Language
Pegasus Schemas
Avro Schemas
+optional record fields, +typerefs
Core schema language:
records, maps, arrays, unions, enums,
primitives
Courier
Why Schemas?
Increase transparency
into the structure of our
data.
With a common
understanding of
structure type-safety can
span multiple languages
and platforms.
Why Pegasus Schemas?
Pegasus schemas have a
rich type system.
Pegasus schemas are
machine readable.
Pegasus schemas are
easy to extend.
Pegasus schemas work
with multiple data
formats.
Pegasus Schema Types
Pegasus Type Scala Type Example JSON
int, long, float, double, boolean,
string
Int, Long, Float, Double, Boolean, String 1, 10000000, 3.14, 2.718281, true, “Coursera”
record case class R(f1: T1, f1: T2, ...) { “f1”: 1, “f2”: “Coursera” }
array A extends IndexedSeq[T] [1, 2, 3]
map M extends Map[String, T] { “key1”: 1, “key2”: 2 }
union sealed abstract class U
case class M1(T1) extends U
case class M2(T2) extends U
{
“org.example.M1”: <T1 Value>
}
enum object E extend Enumeration “SYMBOL”
* unions and typerefs will be covered in more detail later.
.pdsc
(pegasus data schema)
Records
org/example/Note.pdsc{
"name": "Note","namespace": "org.example","doc": "A simple note.","type": "record","fields": [
{ "name": "title", "type": "string" },{ "name": "body", "type": "Body", "optional": true }
]}
Scalacase class Note(title: String, body: Option[Body])
ExamplesNote("reminder", Some(Body(…))) => { "title": "reminder", "body": { … } }Note("reminder", None) => { "title": "reminder" }
Records
Schema{ "name": "WithOptionals", "type": "record", …"fields": [
{ "name": "o1", "type": "string", "optional": true },{ "name": "o2", "type": "string", "optional": true, "default": "b" }{ "name": "o3", "type": "string", "optional": true, "defaultNone": true }
]
Generated Scalacase class WithOptionals(
o1: Option[String],o2: Option[String] = Some("b"),o3: Option[String] = None)
ExamplesWithOptionals(o1 = None) => { "o2": "b" }WithOptionals(o1 = Some("a")) => { "o1": "a", "o2": "b" }WithOptionals(o1 = None, o2 = None) => {}WithOptionals(o1 = None, o3 = Some("c")) => { "o3": "c" }
Optional Fields and Defaults
Collection Types
org/example/NotePad.psdc{
"name": "NotePad", "type": "record", …"fields": [
{"name": "notes","type": { "type": "array", "items": "Note" }
}]
}
Generated Scalacase class NotePad(notes: NoteArray)class NoteArray extends IndexedSeq[Note]
ExampleNotePad(notes = NoteArray(Note(…), …)) =>
{ "notes": [ { "title": "…" }, … ] }
Arrays
org/example/TermWeights.pdsc{
"name": "TermWeights", "type": "record", "namespace": "org.example","fields": [
{"name": "byTerm","type": { "type": "map", "keys": "string", "values": "float" }
}]
}
Generated Scalacase class TermWeights(byTerm: FloatMap)class FloatMap extends Map[String, Float]
ExampleTermWeights(byTerm = FloatMap("ninja" -> 0.25f, "pirate" -> 0.75f)) =>
{ "byTerm": { "ninja": 0.25, "pirate": 0.75 } }
Maps
Unions
Unions are the closest
thing Pegasus has to a
polymorphic type.
Unlike true subtype
polymorphism, unions
have a “sealed” set of
member types.
org/example/Question.pdsc{
"name": "Question", "type": "record", …"fields": [
{ "name": "answerFormat", "type": [ "MultipleChoice", "TextEntry" ] }]
}
Scalacase class Question(answerFormat: Question.AnswerFormat)object Question {
sealed abstract class AnswerFormat()case class MultipleChoiceMember(value: MultipleChoice) extends
AnswerFormatcase class TextEntryMember(value: TextEntry) extends AnswerFormat
}
ExampleQuestion(answerFormat = MultipleChoiceMember(MultipleChoice(…)) =>
{ "answerFormat": { "org.example.MultipleChoice": { … } }
Unions
Enums
org/example/Fruits.pdsc{
"type" : "enum","name" : "Fruits","namespace" : "org.example","symbols" : ["APPLE", "BANANA", "ORANGE"]
}
Generated Scalaobject Fruits extend Enumeration {
val APPLEval BANANAval ORANCE
}
ExampleFruits.APPLE => "APPLE"
Enums
Typerefs
org/example/Timestamp.pdsc{
"type" : "enum","name" : "Timestamp","namespace" : "org.example","ref" : "long"
}
Generated ScalaUse Long directly.
Example1434916256
Basic Typeref
Typerefs are Pegasus’s
multi-tool.
org/example/AnswerFormats.pdsc{
"name": "AnswerFormats", "namespace": "org.example", "type": "typeref","ref": [ "MultipleChoice", "TextEntry" ]
}
Scalasealed abstract class AnswerFormats()object AnswerFormats {
case class MultipleChoiceMember(v: MultipleChoice) extends AnswerFormatscase class TextEntryMember(v: TextEntry) extends AnswerFormats
}
ExampleMultipleChoiceMember(MultipleChoice(…)) =>
{ "org.example.MultipleChoice": { … }
Naming a Union with a Typeref
org/example/DateTime.pdsc{
"name": "DateTime", "namespace": "org.example", "type": "typeref","ref": "string","scala": {
"class": "org.joda.time.DateTime","coercerClass": "org.coursera.models.common.DateTimeCoercer"
}}
ScalaUse org.joda.time.DateTime directly.
ExampleRecord(createdAt = new org.joda.time.DateTime(…)) =>
{ "createdAt": "2015-06-21T18:24:18Z" }
Custom Bindings with a Typeref
Pegasus System
Schema system: Schema based validation + custom validators.
Data system: JSON Object and Array equivalent types, support for binding to native types.
Code generators: Java, Scala (via Courier), Swift (in progress), Android Java (planned)
Codecs:
● via Pegasus:
o JSON - Jackson streaming
o PSON - non-standard JSON equivalent binary protocol
o Avro binary - compact binary protocol
● via Courier:
o StringKeyCodec - compatible with our legacy StringKeyFormats
o InlineStringCodec - a new “URL Friendly” JSON compatible format
Hardened and performance optimized at Linkedin. In large scale production use for over 3 years.
Development with Courier
Courier SBT Plugin
Code generation
integrated into SBT build.
Caches .pdsc file state,
only runs generator when
they change.
Courier SBT Plugin
