la programmation concurrente par flux de données

ACENSI, Tour Monge - 22, Place des Vosges - 92 400 Courbevoie - La Défense 5 - www.acensi.fr

Concurrent programming based on dataflow

TPL DATAFLOW

A new approach to Monte Carlo VAR

09/02/2015Version du document

OVERVIEW

Optimization and multithreading

without getting your hands dirty!

09/02/2015 2Version du document

TPL Dataflow Presentation

Why TPL Dataflow ?

A natural extension of framework 4.0

The library

Use cases

Case study : Monte Carlo Value At Risk (VAR)

What is VAR ?

Monte Carlo VAR: Basic Approach

Monte Carlo VAR: Dataflow Approach

Conclusion

SUMMARY


Speakers Presentation

Yves Alexandre SIMON James KOUTHON Julien LEBOT Adina SANDOU

R&D Director Technical Director .Net Expert .Net Expert

Information systems and Microsoft technologies Consulting

WHO ARE WE ?


Presentation

TPL DATAFLOW


TPL DATAFLOW: A NATURAL EXTENSION OF FRAMEWORK 4.0

Promotes actor-agent oriented designs through primitives.

Allows developers to create blocks to express computations based on directeddataflow graphs.

09/02/2015Version du document 6

TPL DATAFLOW: THE LIBRARY

Overview TPL Dataflow falls in line with Map/Reduce

Can handle large volumes of data

Ideal for long computations

TPL Dataflow: paradigm shift Tasks are created and linked together as a graph

Each node can receive data as input and/or output data



Source blocks (1): acts like a source of data

ISourceBlock<TOutput>

Target blocks (2): acts like a receiver of data

ITargetBlock<TInput>

Propagator blocks: acts like (1) and (2)

IPropagatorBlock<TInput, TOutput>



Basic blocks BufferBlock: is a queue, a FIFO (First In First Out) buffer.

ActionBlock: like a “foreach”, it executes a delegate for each input item.

ex: var node = new ActionBlock<string>(s => Console.WriteLine(s));

TransformBlock: acts like a “Linq” selectex: var node = new TransformBlock<int, int>(p => p * 100);

Advanced blocks BroadcastBlock: forwards copies of data items as its output.

JoinBlock: collects many inputs and output a tuple

Others



Linking Used to link two blocks together.

Predicates and parallelism options available.

There’s no limit to what you can link.

Completion Status Each block supports an asynchronous form of

completion to propagate finished state.


WHY TPL DATAFLOW?

TPL Dataflow benefits

Paradigm shift for higher code expressivity

Using multithreading without effort

Boosting performance (optimization) painlessly

Focusing on the 'what' rather than the 'how'


TPL DATAFLOW: USE CASES

Build more complex systems easilySamples:

Data analysis/mining services

Web-crawlers

Image and Sound processors

Databases engine designs

Financial computation

…


Monte Carlo Value at Risk (VAR)

CASE STUDY


WHAT IS VAR?

What is VAR? Value at risk (VAR)

Monitor risk in trading portfolio

Financial Global risk indicator

Our use case Market VAR (VAR on market move)

Intensive computation (especially for Monte Carlo VAR)


ExampleVAR 99/1D : Maximum lost in 1 day with99% probability

VAR Calculation Methods

Historical VAR

(historicaldata)

ParametricVAR

(formula data)

Monte Carlo VAR (montecarlosimulation

data)

SIMPLE MONTECARLO VAR WORKFLOW


Start

Portfolios Composition

Market Data

Static Data

Global Position

Position Pricing With MonteCarlo

Calculus


Calculus


Calculus

Statistics on Global

Distribution (VAR)End

1 2 3 4

Basic approach

MONTE CARLO VAR


MONTE CARLO VAR: BASIC APPROACH

Pipeline:


StartPortfolios

CompositionMarket Data

Global Position


Calculus


Distribution (VAR)End


Portfolio composition

Fetch portfolios by using the provider

Market data

Get product parameters from market data provider

Global position

Look over all portfolios and nettings and get the positions


Portfolios = PortfolioProvider.Portfolios;

ProductParameters = ProductParametersProvider.ProductsParameters;

Portfolios Composition

Market Data

Global Position

IEnumerable<KeyValuePair<Product, long>> allTransactions = Portfolios.SelectMany(x => x.Transactions)

.GroupBy(y => y.Product).Select(z => new KeyValuePair<Product, long>

(z.Key, z.Sum(x => x.Position)));

Positions = allTransactions.ToDictionary(t => t.Key, t => t.Value);


Position pricing For each product, run the Monte Carlo simulation

Statistics on global Multiply the result by the position value and calculate the lost value


IEnumerable<double> results = StatisticsUtilities.SimulateMonteCarloWithPosition(

new MonteCarloInput{

Parameters = parameters,Position = position,Product = product

}, TotalSimulations);


Calculus

IList<double> totals = new List<double>();

Func<IList<double>, string, IList<double>> sumList = (current, key) => Helpers.SumList(current, lostsValuesByProduct[key].ToList());



totals = lostsValuesByProduct.Keys.Aggregate(totals, sumList);

StatisticsUtilities.CalculateVar(totals, 0.99);

Aggregate the lost value for all products

Choose the VAR at 99% for 1 day


Distribution (VAR)

Dataflow approach

MONTE CARLO VAR


MONTE CARLO VAR: DATAFLOW APPROACH

DataFlow Graph


Portfolios Composition And Market

Data

Global Position


Calculus


Calculus


Calculus

AggregatorStatistics on

Global Distribution (VAR)

DataFlow


Chosen approach: parallelize per product


Product

Product

Product

Product

…

N threads

CalculateLoss() x M iterations





Process overview


TransformBlock

PriceMean

Standard DevPosition

IN: MonteCarloInput OUT: IEnumerable<double>

Losses

Normal distribution

Calculate Loss

ActionBlock TotalsLosses

IN: IEnumerable<double> OUT: IEnumerable<double>

Aggregator


TransformBlock runs the Monte Carlo simulation

Key points:

▬ Do only one thing

▬ Keep work data local

▬ Fully enumerate returned data


var monteCarlo = new TransformBlock<MonteCarloInput, IEnumerable<double>>(input =>

{

var normalDistribution = new NormalEnumerable();

return normalDistribution.Take(TotalSimulations)

.Select(alea => StatisticsUtilities.CalculateLoss(input, alea))

.ToList(); // Very important

}, ExecutionOptions);


Calculus


ActionBlock aggregates the result

No need to synchronize access to shared data


var totals = new List<double>();

var aggregate = new ActionBlock<IEnumerable<double>>(doubles =>

{

if (!totals.Any())

{

totals.AddRange(doubles);

}

else

{

var losses = doubles.ToList();

foreach (var i in Enumerable.Range(0, losses.Count()))

{

totals[i] += losses[i];

}

}

});

Aggregator


Linking the blocks together

Triggering the data flow chain

Data posted asynchronously


foreach (var portfolio in Portfolios

.SelectMany(x => x.Transactions)

.GroupBy(y => y.Product)

.Select(z => new KeyValuePair<Product, long>(z.Key, z.Sum(x => x.Position))))

{

var position = portfolio.Value;

var parameters = ProductParameters.First(x => x.Product.Equals(portfolio.Key));

monteCarlo.Post(new MonteCarloInput

{

Parameters = parameters,

Position = position

});

}

monteCarlo.LinkTo(aggregate, DataflowLinkOptions);

Global Position


Completing the tasks

Tricky to get right

▬ Can cause deadlocks

▬ Solution: Automatically propagate completion


monteCarlo.Complete();

aggregate.Completion.Wait();

DataflowLinkOptions = new DataflowLinkOptions

{

PropagateCompletion = true

}


Manual completion propagation

Maximizing CPU usage


monteCarlo.Completion.ContinueWith(t =>

{

if (t.IsFaulted)

{

((IDataflowBlock)aggregate).Fault(t.Exception); // Pass exception

}

else

{

aggregate.Complete(); // Mark next completed

}

});

ExecutionOptions = new ExecutionDataflowBlockOptions

{

MaxDegreeOfParallelism = Environment.ProcessorCount

}


Result


0

500

1000

1500

2000

2500

3000

3500

4000

i5-4200U 4 @2.30GHz

Intel CeleronG1820 2 @

2.70GHz

Intel i5-2400 4 @3.00GHz

i7-3770K w/ 8 @5.09GHz

i7-4790K w/ 8 @4.00GHz

mill

isec

on

ds

CPU

Benchmark (lower is better)

Basic Data flow

What did we learn?

CONCLUSION


CONCLUSION

Performance increase Faster

Automatically scale to hardware

Paradigm shift Macro-level optimization

New primitives


github.com/acensi/techdays-2015

msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx

github.com/akkadotnet/akka.net

Find out more !

Experiment with the code

Parallelize data loading

Try new blocks

Come see us at the booth

Going further

https://github.com/acensi/techdays-2015

https://msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx

https://github.com/akkadotnet/akka.net

www.acensi.fr

Let’s keep the conversation going!

Come see us at booth 26


la programmation concurrente par flux de données

Technology

dataflowtpl dataflow

var node

documentwhy tpl dataflow

libraryoverview tpl

risk var monitor risk

long computationstpl

basic approachmonte

monte carlo var09022015version