f# streamsanirothan.github.io/streamspresentation/fsharpstreams.pdf · f# streams a lightweight...
TRANSCRIPT
![Page 1: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/1.jpg)
F# StreamsA lightweight F#/C# library for efficient functional-style
pipelines on streams of data.
![Page 2: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/2.jpg)
About MeGian Ntzik (aka Jan Dzik)@anirothanImperial College, Nessos
![Page 3: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/3.jpg)
About NessosISV based in Athens, Greece.NET expertsOpen source F# projects
{m}braceFsPickler, Vagrant, and of course Streams
![Page 4: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/4.jpg)
MotivationMake functional data query pipelines FAST
![Page 5: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/5.jpg)
LinqOptimizerAn automatic query optimizer-compiler for Sequential and
Parallel LINQ.https://github.com/nessos/LinqOptimizer
![Page 6: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/6.jpg)
LinqOptimizercompiles LINQ queries into fast loop-based imperativecodespeedups of up to 15x
![Page 7: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/7.jpg)
ExampleThe query
var query = (from num in nums.AsQueryExpr() where num % 2 == 0 select num * num).Sum();
compiles toint sum = 0;for (int index = 0; index < nums.Length; index++){ int num = nums[index]; if (num % 2 == 0) sum += num * num;}
![Page 8: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/8.jpg)
DisadvantagesRuntime compilation
Overhead (mitigated by caching)Emitting IL not cross-platform (e.g. security restrictionsin cloud, mobile)Access to private fields/methods?
New operations => compiler changesProblematic F# support
![Page 9: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/9.jpg)
Should become a Roslyn compile time plugin in future
![Page 10: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/10.jpg)
Clash of the LamdasICOOOLPS'14
Aggelos Biboudis (@biboudis)
Nick Palladinos (@NickPalladinos)
Yannis Smaragdakis
![Page 11: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/11.jpg)
Performance BenchmarksSum (windows)
![Page 12: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/12.jpg)
Sum (linux)
![Page 13: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/13.jpg)
Sum of squares (windows)
![Page 14: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/14.jpg)
Sum of squares (linux)
![Page 15: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/15.jpg)
Sum of even squares (windows)
![Page 16: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/16.jpg)
Sum of even squares (linux)
![Page 17: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/17.jpg)
Cartesian product (windows)
![Page 18: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/18.jpg)
Cartesian product (linux)
![Page 19: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/19.jpg)
Java 8 very fast
![Page 20: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/20.jpg)
LinqOptimizer improving F#/C# performance
![Page 21: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/21.jpg)
What makes Java 8 faster?
![Page 22: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/22.jpg)
Streams!
![Page 23: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/23.jpg)
Typical Pipeline Pattern1: source |> inter |> inter |> inter |> terminal
inter : intermediate (lazy) operations, e.g. map, filterterminal : produces result or side-effects, e.g. reduce, iter
![Page 24: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/24.jpg)
Seq example1: 2: 3: 4: 5:
let data = [| 1..10000000 |] |> Array.map int64data|> Seq.filter (fun i -> i % 2L = 0L) //lazy|> Seq.map (fun i -> i + 1L) //lazy|> Seq.sum //eager, forcing evaluation
![Page 25: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/25.jpg)
Seq is pulling1: 2: 3: 4: 5:
let data = [| 1..10000000 |] |> Array.map int64data|> Seq.filter (fun i -> i % 2L = 0L) //lazy inter|> Seq.map (fun i -> i + 1L) //lazy inter|> Seq.sum //eager terminal, forcing evaluation
The terminal is pulling data from the pipeline viaIEnumerator.Current and IEnumerator.MoveNext()
![Page 26: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/26.jpg)
With Streams1: 2: 3: 4: 5:
let data = [| 1..10000000 |] |> Array.map int64Stream.ofArray data //source|> Stream.filter (fun i -> i % 2L = 0L) //lazy|> Stream.map (fun i -> i + 1L) //lazy|> Stream.sum //eager, forcing evaluation
![Page 27: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/27.jpg)
Streams are pushing!
![Page 28: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/28.jpg)
Streams are pushing1: 2: 3: 4:
Stream.ofArray data //source|> Stream.filter (fun i -> i % 2L = 0L) //lazy|> Stream.map (fun i -> i + 1L) //lazy|> Stream.sum //eager, forcing evaluation
The source is pushing data down the pipeline.
![Page 29: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/29.jpg)
How does it work?
![Page 30: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/30.jpg)
Starting from Seq.iter1: Seq.iter : ('T -> unit) -> seq<'T> -> unit
![Page 31: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/31.jpg)
Flip the args1: seq<'T> -> ('T -> unit) -> unit
![Page 32: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/32.jpg)
Stream!1: type Stream<'T> = ('T -> unit) -> unit
![Page 33: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/33.jpg)
Continuation passing style!
![Page 34: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/34.jpg)
Let's make us some (simple) Streams!
![Page 35: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/35.jpg)
Simple Streams1: type Stream = ('T -> unit) -> unit
Can do map, filter, fold, iter
![Page 36: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/36.jpg)
When to stop pushing?1: type Stream = ('T -> unit) -> unit
Stopping push required for e.g.1: Stream.takeWhile : ('T -> bool) -> Stream<'T> -> Stream<'T>
![Page 37: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/37.jpg)
Stopping pushChange
1: type Stream = ('T -> unit) -> unit
to1: type Stream = ('T -> bool) -> unit
![Page 38: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/38.jpg)
What about zip?1: Stream.zip : Stream<'T> -> Stream<'S> -> Stream<'T * 'S>
Zip needs to synchronise the flow of values.Zip needs to pull!
![Page 39: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/39.jpg)
Streams can push and pull1: 2: 3: 4: 5: 6: 7:
// ('T -> bool) is the composed continutation with 'T for the current value // and bool is a flag for early termination// (unit -> unit) is a function for bulk processing// (unit -> bool) is a function for on-demand processing
/// Represents a Stream of values.type Stream<'T> = Stream of (('T -> bool) -> (unit -> unit) * (unit -> bool))
![Page 40: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/40.jpg)
The Streams libraryImplements a rich set of operations
![Page 41: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/41.jpg)
More examples
![Page 42: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/42.jpg)
Parallel Streams1: 2: 3: 4: 5: 6:
let data = [| 1..10000000 |] |> Array.map int64data|> ParStream.ofArray|> ParStream.filter (fun x -> x % 2L = 0L)|> ParStream.map (fun x -> x + 1L)|> ParStream.sum
![Page 43: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/43.jpg)
Cloud Streams!Example: a word count
![Page 44: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/44.jpg)
Streams are lightweight and powerfulIn sequential, parallel and distributed flavors.
![Page 45: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/45.jpg)
The holy grail is in reachWe can write functional pipelines with the performance of
imperative code.Stream fusion: from lists to streams to nothing at all, Duncan
Coutts, Roman Leshchinskiy, and Don Stewart, ICFP '07
![Page 46: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/46.jpg)
AlmostDepends on the compiler's ability to inline.
![Page 47: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/47.jpg)
Inlining continuations = stream fusion
![Page 48: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/48.jpg)
Stream operations are non-recursiveIn principal, can be always fused (in-lined).
Not always done by F# compiler.
![Page 49: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/49.jpg)
Experiments with MLtonby @biboudis
https://github.com/biboudis/sml-streamsMLton appears to always be fusing.
![Page 50: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/50.jpg)
Can we make the F# compiler smarter?
![Page 51: F# Streamsanirothan.github.io/StreamsPresentation/FSharpStreams.pdf · F# Streams A lightweight F#/C# library for efficient functional-style pipelines on streams of data. About Me](https://reader033.vdocuments.net/reader033/viewer/2022042809/5f955e0877ae843050553d73/html5/thumbnails/51.jpg)
Questions?