formal verification of storm topologies through...

DICEHorizon2020Research&InnovationActionGrantAgreementno.644869http://www.dice-h2020.eu

FundedbytheHorizon2020FrameworkProgrammeoftheEuropeanUnion

FormalVerificationofStormTopologiesthroughD-VerT

SA-TTAatSAC2017,MarrakechApr4th,2017

FrancescoMarconi,MarcelloM.BersaniandMatteoRossiDEIB, Politecnico di Milano, Italy

Roadmap

§ ContextandMotivation• StreamingApplication• ApacheStorm• DICEProject

§ D-VerT• Overview• Workflow• Usecase

§ Conclusion

2

CONTEXTANDMOTIVATIONFormalVerificationofStormTopologiesthroughD-VerT

3

Data-IntensiveApplications(DIAs)

o Needtoprocessdatabeing§ Massivelylargeinsize§ Complex§ Rapidlychanging

o DevotemostoftheirprocessingtimetoI/O,movementandmanipulationofdata.

o Relyonso-called"Bigdatatechnologies”

4

StreamingApplications

o SpecialcaseofDIAso Needtoprocessacontinuousflowofinformation

§ Streamà unboundedsequenceoftuples(messages)

o Usuallydescribedbymeansofatopology§ Graphofcomputationscomposedof

• input nodes(sourceofdatastreams)• computational nodesàmanipulatedatastreams

o Calculate,Filter,Aggregate,Join,Talktodatabases,etc

5

QualityIssuesinStreamingDIAso Importantrequirementsforstreamingapplications

§ Latency§ Throughput

o Criticalpoints§ incorrectdesignoftimingconstraints§ nodefailures

o mightcause§ Highlatencyinprocessingtuples§ Memorysaturation

6

latency

throughput

ApacheStorm

o OpenSourceDistributedStreamProcessingSystemo Analytics,LogEventprocessing,etc..o Reliability,at-least-onesemanticso Wideadoptioninproductiono InStormtopologies

§ Inputnodescalledspouts§ Computationalnodescalledbolts

7

DICEProject

o Horizon2020Research&InnovationAction(RIA)§ Quality-AwareDevelopmentforData-Intensiveapplications§ Feb2015- Jan2018,4MEurosbudget§ 9partners(Academia&SMEs),7EUcountries

8

Platform-IndependentModel

TheDICEApproach

9©DICE

Platform-SpecificModel

DICE profile

Deployment&ContinuousIntegration

DICE IDE

Big Data

ContinuousMonitoring

DeploymentModel

QAModels

ContinuousValidation

FormalVerification

D-VerT- DICEVerificationtool

o Assessmentofnon-functionalpropertiesof(streaming)DIAsatdesigntime

o Reliesonformalmodelingofstreamingapplications§ TemporalLogicModel(CounterNetworks)

o Aimsatcheckingthepresenceoftemporaldesignflaws§ Boltsnotabletoprocesstheincomingflowofmessages

10

D-VerTworkflow

11

- Create UML Model- Apply DICE Profile- Define Topology Structure- Apply Stereotypes- Provide values to parameters

MODEL

Run Verification

VERIFY

DICE-Profiled UML

Tool Configuration

Formal Model

TRANSFORM

✅

!REFINE

Property: “allbolts’inputbuffershaveaboundedoccupationlevel”

+

📈

Usecase:WebCrawler

o Goal:developawebcrawlingapplication§ Tofetch,parseandindexwebresources

• Input:continuousstreamofURLs• Output:indexedwebresources

§ Streamingparadigmwellsuitedforthistask

12

Usecase:WebCrawlerPlatform-IndependentModel(PIM)

13

o Goal:developawebcrawlingapplication§ Tofetch,parseandindexwebresources

• Input:continuousstreamofURLs• Output:indexedwebresources

§ Streamingparadigmwellsuitedforthistask

URLsStreamSource<<SourceNode>>

IndexingNode<<StorageNode>>

WebCrawlingApplication

<<ComputationNode>>

Usecase:WebCrawlerPlatform- SpecificModel(1)

14

Usecase:WebCrawlerPlatform-SpecificModel- Refinement

15

Usecase:WebCrawlerOutputtrace

16

CONCLUSIONS 17

Wrapup

oModel-drivenapproachtoformalverificationofstreamingapplications

o ApplicationdesignedandrefinedasannotatedUMLdiagrams§ SupportedbyD-VerTtool

• Basedontemporallogicmodel• automaticallygeneratesformalmodelsfromUMLmodelsandrunverification

• Checkwhetheranyofthebolts’queuesgrowsindefinitely

18

Futureworks

o Identificationandverificationoffurtherproperties

oModelingdifferenttechnologies§ Spark,CEP,…

o Toolandmodelimprovements

19

Questions?

20

Thankyou

21