azure data factory for beginners - darius liktorius · my background cloud architect with cognizant...
TRANSCRIPT
![Page 1: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/1.jpg)
Azure Data Factory
For BeginnersDARIUS LIKTORIUS
![Page 2: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/2.jpg)
Please Silence
Your Devices
![Page 3: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/3.jpg)
Contact & Follow Me
Presentation Landing Page & Resources:
Liktorius.com/go/ADFBeginners
Twitter: @DLiktorius
Blog: Liktorius.com
Follow me on LinkedIn:linkedin.com/in/DariusLiktorius
![Page 4: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/4.jpg)
Introductions
![Page 5: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/5.jpg)
My Background
Cloud Architect with Cognizant Softvision
Founder of NetCorp – since 1997
Using and Certified in SQL Server since 6.5
SQL: Architecture, Development, Infrastructure & DBA
Certified on both AWS and Azure
MCSE, MCSD, MCDBA & MCT – since 1996
Hobby: Professional Photography
![Page 6: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/6.jpg)
About You
DBA’s?
DB or BI Developers?
Architects?
Who’s used SQL in the Azure?
![Page 7: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/7.jpg)
Our Agenda
Intro to Azure Data Factory (ADF)
ADF Components
Demo: Creating a Data Factory
Demo: Creating and Debugging a Pipeline
Demo: Triggers & Monitoring
ADF Pricing
Q&A (?)
![Page 8: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/8.jpg)
Intro to
Azure Data Factory
(ADF)
![Page 9: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/9.jpg)
What is Azure Data Factory?
Microsoft Azure Platform as a Service (PaaS) offering
Runs in the Cloud but Hybrid with on-prem features
Data Integration Service (remember SSIS?)
ETL (Extract Transform and Load)
ELT (Extract Load and Transform)
Create pipelines to Copy and Transform your data
![Page 10: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/10.jpg)
Azure Data Factory
Components
![Page 11: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/11.jpg)
Components of Azure Data Factory
Data Factory
Pipelines
Activities
Data Flows
Datasets
Linked Services
Integration Runtimes
Triggers
Templates
![Page 12: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/12.jpg)
ADF Components – Data Factory
Environmental Boundary
Container to hold all other ADF components
Deploy as many as you want/need (Dev, QA, Prod)
Security (IAM) Boundary
Integrates with Git and Azure DevOps
Deployable as a single ARM template
![Page 13: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/13.jpg)
ADF Components – Pipelines
Definition of executable ETL/ELT workflow
Similar to an SSIS package
Canvas designer interface
Collection of units of work (Activities)
Create new or from a Template
![Page 14: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/14.jpg)
ADF Components – Activities
Single unit of work / task
Chained in sequence or Parallelized
Multiple categories:
Move & transform
Iteration & conditionals
Azure Function, Databricks, Machine Learning
More…
![Page 15: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/15.jpg)
ADF Components – Data Flows
Recently introduced
Separate type of activities with their own canvas
Mapping Data Flows – Transform Data
Wrangling Data Flows (Preview) – Prepare Data
Executed from a Pipeline
Run on Databricks and Spark environments
![Page 16: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/16.jpg)
ADF Components – Datasets
Data reference objects
Define format of input/output data
Represent single table/view, file or folder
![Page 17: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/17.jpg)
ADF Components – Linked Services
Data source / service connection object
Ninety (90) supported services – and growing
From SQL Server to Cosmos DB to MySQL to Oracle
From Amazon S3 to SAP to SFTP to QuickBooks
Maintains connection and credential properties
![Page 18: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/18.jpg)
ADF Components – Integration Runtimes
Reference infrastructure to run Activities on
Three types:
Azure
Self-Hosted
Azure-SSIS
![Page 19: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/19.jpg)
ADF Components – Triggers
Define when to execute your Pipelines
Use a recurring schedule, interval or event based
![Page 20: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/20.jpg)
ADF Components – Templates
Created from your own Pipeline
Samples available from Microsoft
![Page 21: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/21.jpg)
ADF Component Summary
Credit: cathrinewilhelmsen.net
![Page 22: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/22.jpg)
DEMO
Creating a
Data Factory
![Page 23: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/23.jpg)
DEMO
Creating &
Debugging
Pipelines
![Page 24: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/24.jpg)
DEMO
Triggers &
Monitoring
![Page 25: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/25.jpg)
Azure Data Factory
Pricing
![Page 26: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/26.jpg)
Azure Data Factory - Pricing
Prices vary based on Azure vs Self-hosted Runtime
Orchestration charges (per 1,000 runs)
Execution charges (per hour, by activity type)
Data Flows (per vCore-hour)
Data Factory Operations
Read/Write: per 50k modified/referenced entities
Monitoring: per 50k run records retrieved
![Page 27: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/27.jpg)
Q&ADon’t be shy!
![Page 28: Azure Data Factory For Beginners - Darius Liktorius · My Background Cloud Architect with Cognizant Softvision Founder of NetCorp –since 1997 Using and Certified in SQL Server since](https://reader035.vdocuments.net/reader035/viewer/2022062506/5f01b66d7e708231d400addd/html5/thumbnails/28.jpg)
Contact & Follow Me
Presentation Landing Page & Resources:
Liktorius.com/go/ADFBeginners
Twitter: @DLiktorius
Blog: Liktorius.com
Follow me on LinkedIn:linkedin.com/in/DariusLiktorius