making magic with f# type providers
DESCRIPTION
Combining Data, Information, Services and Programming, at Internet-Scale Presented at NDC London December 2013 Dr. Don Syme Follow @dsyme on TwitterTRANSCRIPT
![Page 1: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/1.jpg)
| Basel
Making Magic with F# Type ProvidersCombining Data, Information, Services and Programming, at Internet-Scale
Dr. Don SymePrincipal Researcher, Microsoft, F# Community Contributor@dsyme
![Page 2: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/2.jpg)
Proposition 1The world is information-rich
![Page 3: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/3.jpg)
The Information Revolution
1 2 3 4 5 6 7
![Page 4: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/4.jpg)
Proposition 2Modern programming is
intenselyinformation-rich
![Page 5: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/5.jpg)
Proposition 3Our languages are information-
sparse
![Page 6: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/6.jpg)
The developer’s perspective
Languages do not integrate information
Non-intuitiveNot simpleDisorganisedStaticHigh friction
![Page 7: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/7.jpg)
We need to bring information into the language…
At internet-scale, strongly tooled, strongly typed
![Page 8: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/8.jpg)
Paradigm Locator
Statically Typed
Dynamically Typed
![Page 9: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/9.jpg)
Paradigm Locator
Statically Typed
Dynamically Typed
A major search is on!
• make statically typed langs more dynamic
• make dynamically typed langs more static
• moderate static typing in limited ways
![Page 10: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/10.jpg)
But before we get into that…
A bit about F#
![Page 11: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/11.jpg)
F# is Open Source, is Cross Platform
fsharp.orgfsharp.org/use/macfsharp.org/use/linuxfsharp.org/use/androidfsharp.org/use/iosfsharp.org/use/windowsfsharp.org/use/freebsd
![Page 12: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/12.jpg)
A strongly typed, data-immersive functional-first
language for….
simple code
![Page 13: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/13.jpg)
let swap (x, y) = (y, x)
let rotations (x, y, z) = [ (x, y, z); (z, x, y); (y, z, x) ]
let reduce f (x, y, z) = f x + f y + f z
C#Tuple<U,T> Swap<T,U>(Tuple<T,U> t){ return new Tuple<U,T>(t.Item2, t.Item1)}
ReadOnlyCollection<Tuple<T,T,T>> Rotations<T>(Tuple<T,T,T> t)
{ new ReadOnlyCollection<int> (new Tuple<T,T,T>[] {new Tuple<T,T,T>(t.Item1,t.Item2,t.Item3); new Tuple<T,T,T>(t.Item3,t.Item1,t.Item2); new Tuple<T,T,T>(t.Item2,t.Item3,t.Item1); });}
int Reduce<T>(Func<T,int> f,Tuple<T,T,T> t) { return f(t.Item1)+f(t.Item2)+f(t.Item3); }
F#
![Page 14: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/14.jpg)
type Rating = { UserId: int; DishId: int; Rating: int }
![Page 15: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/15.jpg)
data.Countries.Brazil.Indicators.``Population ages 0-14 (% of total)``
![Page 16: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/16.jpg)
10 Reasons Not to Use a Strongly Typed Functional Language
![Page 17: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/17.jpg)
Example (power company)I have written an application to balance the national power generation schedule for a portfolio of power stations to a trading position for an energy company. ...the calculation engine was written in F#.
The use of F# to address the complexity at the heart of this application clearly demonstrates a sweet spot for the language within enterprise software, namely algorithmically complex analysis of large data sets.
Simon Cousins (power company)
fsharp.org/testimonials
![Page 18: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/18.jpg)
F# at KaggleAt Kaggle we initially chose F# for our core data analysis algorithms because of its expressiveness.
We’ve been so happy with the choice that we’ve found ourselves moving more and more of our application …into F#.
The F# code is shorter, easier to read, easier to refactor, and, because of the strong typing, contains far fewer bugs.
As our data analysis tools have developed, we’ve seen domain-specific constructs emerge very naturally
As our codebase gets larger, we become more productive.
fsharp.org/testimonials
![Page 19: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/19.jpg)
F# is changing…
F# runs on many
platforms
“F# is for Windows”
![Page 20: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/20.jpg)
OverviewF# is changing…
“F# has many
contributors”
“Microsoft makes F#”
![Page 21: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/21.jpg)
OverviewF# is changing…
Many perspective
shttp://fsharp.org
One perspective(Microsoft’s)
http://msdn.microsoft.com
![Page 22: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/22.jpg)
F# Language
Visual F# Tools
Visual Studio, ..NET, SDKs, Tools
fssnip.net
tryfs.org
fsharpforfunandprofit.com
tryfs.org
Xamarin Studio
fpish.net
F# Open SourceGroup
F# Community Groups
tryfsharp.orgF# Software Foundation
…New think…
![Page 23: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/23.jpg)
On Windows, Microsoft recommend
F# and the Visual F# toolsfor your functional-first
programming needs
![Page 24: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/24.jpg)
Functional-first programming is a general-purpose programming technique particularly suited to tasks where Time-to-deployment, Efficiency, Correctness and Taming Complexity dominate.
![Page 25: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/25.jpg)
Examples include ETL pipelines, general data-manipulation, calculation engines, programmatic UIs and data science.
While these problems can be solved using other programming paradigms, they are particularly amenable to functional-first programming.
![Page 26: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/26.jpg)
Functional-first programming uses functional programming as the initial paradigm for most purposes, but employs other techniques such as objects and state as necessary.
![Page 27: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/27.jpg)
Give your
world some F# spice!
iOS
![Page 28: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/28.jpg)
![Page 29: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/29.jpg)
Exosystem: Cloud Programming
![Page 30: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/30.jpg)
Amazon Web Services .NET SDKs
http://aws.amazon.com/tools
https://github.com/aws (Apache 2.0)
![Page 31: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/31.jpg)
Azure .NET SDKs
http://www.windowsazure.com
https://github.com/WindowsAzure/ (Apache 2.0)
![Page 32: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/32.jpg)
Learn morefsharp.orgfsharp.org/math fsharp.org/data-science fsharp.org/machine-learningfsharp.org/cloudfsharp.org/data-access
fsharp.net (MSDN pages)
![Page 33: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/33.jpg)
Back to the main topic…
![Page 34: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/34.jpg)
We need to bring information into the language…
At internet-scale, strongly tooled, strongly typed
![Page 35: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/35.jpg)
Demo: Integrate all of freebase.com
“as if it were a library”
40M entities, 1Billion facts, 24,000 types, 65,000 properties
![Page 36: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/36.jpg)
Demo: F# to WorldBank
![Page 37: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/37.jpg)
A Type Provider is….
“Just like a library”
“A design-time component that computes a space of types and methods on-demand…”
“An adaptor between data/services and the .NET type system…”
“On-demand, scalable compile-time provision of type/module definitions…”
![Page 38: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/38.jpg)
Data is like water…
![Page 39: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/39.jpg)
Data is like water…
• Everyone needs it. Everyone knows where to get it.• Nobody is sure where it really came from, or goes
to.• …really knows its true cost, or true value.• …likes to pay for it, or to share it.• …knows how much is wasted.• You might get washed away by it.• You only find out it was bad after you have drunk it.
![Page 40: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/40.jpg)
Theme #1
On-Demand Types = Internet Scalable Magic
![Page 41: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/41.jpg)
On-Demand Type Provisionlet data = Freebase.GetDataContext()
1. Compiler/IDE requests metadata for symbol GetDataContext Provider reports return type of FreebaseDataContext
data.
2. Compiler/IDE requests contents of type FreebaseDataContext Provider asks Freebase metadata service for top-level domains Provider reports top-level domains of Freebase as properties of the
type
data.Society
3. Compiler/IDE requests metadata for symbol Society...
![Page 42: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/42.jpg)
Theme #2
Many Data Sources, One Mechanism
![Page 43: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/43.jpg)
All your types are belong to us….
types
![Page 44: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/44.jpg)
SQL
11/04/2023 44
![Page 45: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/45.jpg)
CSV
11/04/2023 45
![Page 46: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/46.jpg)
JSON
11/04/2023 46
![Page 47: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/47.jpg)
XML
11/04/2023 47
![Page 48: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/48.jpg)
OData
11/04/2023 48
![Page 49: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/49.jpg)
Hadoop/Hive
11/04/2023 49
![Page 50: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/50.jpg)
World Bank
11/04/2023 50
![Page 51: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/51.jpg)
Azure Data Market
11/04/2023 51
![Page 52: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/52.jpg)
WSDL
11/04/2023 52
![Page 53: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/53.jpg)
WMI
11/04/2023 53
![Page 54: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/54.jpg)
Freebase
11/04/2023 54
![Page 55: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/55.jpg)
JavaScript/TypeScript
11/04/2023 55
![Page 56: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/56.jpg)
R
11/04/2023 56
![Page 57: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/57.jpg)
DemoF# + Xenomorph TimeScape
An F# type provider for deep, robust integration of financial data
![Page 58: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/58.jpg)
Theme #3
Data and Types at Multiple Scales
![Page 59: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/59.jpg)
Data at Multiple ScalesFrom Everything to Individuals
data.AllEntites
Data Scripters need to work with different
granularities of schematization
data.Automotive.``Automobile Models``
data.Automotive.``Automobile Models``.Individuals.``Porsche 911``
…Only a language that supports massively scalable metadata can operate at all
these levels
Every stable entity can get a unique type
![Page 60: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/60.jpg)
Providing Units of Measurevia F#’s Units of Measure If the metadata contains
units…
…then these can be projected into the
programming language.
![Page 61: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/61.jpg)
Theme #4
Schema Change
![Page 62: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/62.jpg)
A problem, but…
Many, many data sources are surprisingly stable
Some data sources support “snapshot dates”
F# supports “invalidation signals” from providers at design-time
Erasure makes compiled code much less fragile
![Page 63: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/63.jpg)
FSharp.Data
fsharp.github.io/FSharp.Data
on NuGet, use it in Visual Studio today
![Page 64: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/64.jpg)
In Summary Open, cross-platform,
strongly typed, efficient, rock-solid
stable
The safe choice for enterprise data programming
Unbeatable data integration
Visual F# - tooling you can trust from
Microsoft
F#
![Page 65: Making Magic with F# Type Providers](https://reader036.vdocuments.net/reader036/viewer/2022081519/554f5a1db4c905c8088b45b0/html5/thumbnails/65.jpg)
Questions?
Give your .NET
an F# edge!