introduction to d programming language at weka.io
TRANSCRIPT
1
D in real life
By: Liran Zvibel, Weka.IO CTO
2D in real life
• Defining the future of software defined, scale out storage for the cloud-‐based datacenter • VC backed company (NVP, Gemini), largest round-‐A of 2014 • Large, strong engineering team, many XIV veterans • Founded by Omri Palmon, Maor Ben-‐Dayan, Liran Zvibel
About Weka.IO
• Extremely reliable • High performance data path • Complicated “control plane”/“Management code” • Distributed nature due to HA requirements • Low level interaction with HW devices • Some kernel-‐level code, some assembly • Language has to be efficient to program, and fit for large projects
3
Storage system requirements
D in real life
• C codebase • A lot of auto-‐generated code from XML for RPC, clustering code and external APIs (CLI, GUI) - Requires a complicated build process - Difficult to understand where “magic” code came from • Our own implementation of Classes/polymorphism and templates
4
What did we do at XIV?
D in real life
• Data-‐path in C++, control path in Python (or Java, C#, Scala, etc) - Deciding whether to run in the same process or via RPC, both options have downsides
• All in C++ - Outside the scope of this talk, C++ has many downsides writing very large projects, many pitfalls
• Google Go - Can’t disable GC, no generic programming, inheritance, no C ABI. Not performant • Mozilla Rust - Currently a moving target
5
Some other possible options
D in real life
6D in real life
“Often, programming teams will resort to a hybrid approach, where they will mix Python and C++, trying to get the productivity of Python and the performance of C++. The frequency of this approach indicates that there is a large unmet need in the programming language department.”
“D intends to fill that need. It combines the ability to do low-level manipulation of the machine with the latest technologies in building reliable, maintainable, portable, high-level code. D has moved well ahead of any other language in its abilities to support and integrate multiple paradigms like imperative, OOP, and generic programming.”
— Walter Bright, Co-designer of D
• Created ’99 by Walter Bright, Andrei Alexandrescu, as a C++ replacement
• Work on D2 started 2007, considered stable in late 2010.
• Modern programming language with native efficiency, but leveraging 20 years of experience making mistakes while designing C++ and Java.
• Combines the power of C++, but with the usability of languages like Python, Java and C#.
• Has several compilers : dmd, gdc, ldc
7
What is D?
D in real life
• Smaller community • Stability reminds C++ of the early 2000s • Binary compatible to C, no source compatibility - We must “translate” h files to d - Inability to inline c code (that would otherwise be inlined in C++) • Compilation time grows if many templates are instantiated across different compilation units • Missing keyword arguments when calling functions (a la python)
8
D down sides
D in real life
• range is the D equivalent of an iterable • InputRange requires 3 basic methods: - bool empty; - T front; - void popFront(); • foreach(elem; range) • foreach(k, v; dict) • foreach(idx, ref e; array)
9
Ranges and Algorithms
D in real life
Output copy(Input, Output)(ref Input source, ref Output sink) if (isInputRange!Input && isOutputRange!(Output, ElementType!Input)) { foreach(c; source) sink.put(c); return sink; }
10
Simple Algorithm
D in real life
Symbol[] getSymbols(string filename) { return executeShell("nm -S " ~ filename) .output .splitLines .map!” a.split()" .filter!"4 == a.length" .map!((x) => new Symbol(x)) .filter!`a.typeFits(["d", "D", "v", "V"])` .array; }
11
UFCS and Pipeline programming
D in real life
bool binarySearch(T)(T[] input, T value) { while (!input.empty) { auto i = input.length / 2; auto mid = input[i]; if (mid > value) input = input[0 .. i]; else if (mid < value) input = input[i + 1 .. $]; else return true; } return false; }
12
Slices
D in real life
• It supports raw pointers if required • Has native C ABI so linking with existing libraries is easier than C++ • Supports static local variables • Has Templates (Generic programming) - They are much easier and flexible than in C++ - D supports compile time execution — CTFE • Supports Class hierarchies and polymorphism • Statically typed, but with very strong type inference • Supports operator overloading • Inline assembler support
13
D is similar to C++
D in real life
• It supports Garbage Collection - this is the default, but you can opt-‐out, do pointer arithmetic - Compiler helps you verify GC is indeed not being used - Allows prototyping in a high level way, then optimizing GC out where needed
• It has inner-‐classes • It has delegates • Vast support for attributes • Very strong type introspection/reflection allowing “duck typing”
14
Python that compiles?
D in real life
@StatsCategory(“fs_driver_user"):
@RateQuery @StatUnit("Ops/Sec") { @("Number of read operations per second") IncrementalStat READS;
@("Number of write operations per second") IncrementalStat WRITES; } @StatUnit("Microseconds", 0.1) { @("Average latency of READ operations") @MeasuredQuery!`{READS}` TimerStat READ_LATENCY;
@("Average latency of WRITE operations") @MeasuredQuery!`{WRITES}` TimerStat WRITE_LATENCY }
15
Attributes usage
D in real life
• It has modules, packages and imports system • Native Array, Tuple and Associative Array types with bounds checking • Supports nested functions and closures - Also very strong anonymous functions/delegates (lambda with a closure) • Has pure (enforced!) functional programming subset - Higher order functions: reduce, map, filter, compose, curry - supports lazy evaluation of function parameters
16
Python that compiles?
D in real life
auto dumpField(U)(U field) { static if (__traits(hasMember, field, "toJSON")) { return field.toJSON(); } else static if (is(typeof({auto x = JSONValue(field);}))) { return field; } else static if (is(typeof(structToJSON!U))) { return structToJSON(field); } else static if (isArray!U) { return field.map!(dumpField!(typeof(field[0]))).array; } else static if (is(typeof({text(field);}))) { return text(field); } else { return "<unjsonable>"; } }
17
Introspection
D in real life
JSONValue structToJSON(T)(ref T obj) if (is(T == struct) || is(T == class)) { string[string] dummy; auto jv = JSONValue(dummy);
foreach(i, U; typeof(obj.tupleof)) { enum NAME = __traits(identifier, obj.tupleof[i]); jv.object[NAME] = dumpField(obj.tupleof[i]); } return jv; }
18
Introspection
D in real life
struct RobinHashTable(K, V, size_t maxLength) { static if (NumEntries < ushort.max-1) { alias CellIdx = ushort; } else { alias CellIdx = uint; } static if (K.sizeof % 8 < 7) { align(8) struct KV { align(1): K k; ubyte cellData; align(8): V v; } }
19
Compile time magic
D in real life
else { align(8) struct KV { K k; align(8): V v; align(1): ubyte cellData; } }
• All variables are initialized to default values • Contract programming • Casts are explicit, with special keyword • Exceptions support • mixin support • Concurrent programming support - native thread local storage - native locks and message passing mechanisms
20
D is for real SW engineers
D in real life
• static if • scope(exit/success/failure) statement • Unit testing support • -‐debug and -‐release default support • Versioned compilation (special debug case) • @safe/@trusted/@system code • Documentation comments
21
D is for real SW engineers
D in real life
void testing(T...)(T values) { writeln("Called with ", values.length, " arguments."); // Access each index and each value foreach (i, value; values) { writeln(i, ": ", typeid(T[i]), " ", value); } } void main() { testing(5, "hello", 4.2); } Called with 3 arguments. 0: int 5 1: immutable(char)[] hello 2: double 4.2
22
(Hetero) Variadic functions
D in real life
string makeHammingWeightsTable(string name, uint max = 255) { string result = "immutable ubyte[%s] %s = [ ".format(max+1,name); foreach (b; 0 .. max + 1) { result ~= to!string(bitsSet(b)) ~ ", "; } return result ~ "];"; } mixin(makeHammingWeightsTable("hwTable")); unittest { assert(hwTable[10] == 2); assert(hwTable[0] == 0); assert(hwTable[255] == 8); }
23
Mixin, CTFE, Unittests
D in real life
Questions?Peta Exa Zetta Yotta Xenna Weka (1030)
• Tracing system - space and cpu efficient - keep track of func enter/exit - Easy way to keep logs • RPC - Very efficient - Easy to use and declare • CLI • Statistics • Events (Telemetry)
25
Infrastructure
D in real life