deep dive data management gateway - sqlsaturday edinburgh
TRANSCRIPT
Data Management Gateway
Deep Dive
Say Thank you to Volunteers:
They spend their FREE time to give you this event.
Because they are crazy. Because they want YOU
to learn from the BEST IN THE WORLD.
Jean-Pierre RiehlPractice Manager Data & BI
@djeepy1 - http://blog.djeepy1.net
Who am I ?
Pure-Player Microsoft Self-Service BI experts « Agile BI » projects Data Strategy Hybrid-BI
MVP SQL Server
French PASS Chapter Leader
La communauté Data Microsoft
Webcasts, Conférences, Afterworks
.Pro
Save the DateSeptember, 5thCampus SUPINFOTour Montparnasse, Paris
Self-promotion
An hybrid story…
Microsoft Cloud-First strategy…
But your data remains On-Prem…
Here it comes DMG
DMG-aware tools
Power BI for O365
PowerBI.com
Azure Data Factory
Power BI
ExcelPower Query
Power Pivot
Power View Power Map
Power BI Tenant SQL
Cloud On-Prem
Oracle
…Power BI Admin Center
Data Catalog
Q&A
Power BI Sites
Data Refresh
Indexing / Search
OData Access
PowerBI.com
Live Power BI reports &
dashboards
LiveQuery
Scheduled Refresh
Cloud
On-premises
Azure Data Factory
Hybrid features
Data Refresh Data Movement Live Query Indexing Sources OData Access (Data Virtualization)
One DMG to rule them all
DMZ
SQL
Cloud On-Prem
Oracle …
Power BI for O365
DataManagement
Gateway
OnPrem Data Sources
PowerBI.com
Azure Data Factory
Excel
DMG DETAILS
Versions of DMG
There are multiple versions of DMG
Data Management Gateway Power BIADF 1.4.5549.1 24/3/2015
Analysis Services Connector Power BI v2 1.4.5576.2 17/4/2015
Breaking News: 1.4.5612.1 (released 26/5)
What’s that
A Windows Service DIAHostService
A configuration application
Managed by the cloud Power BI Admin Center Azure Portal (new version)
demoFIRST LOOK AT DMG
Data Sources SupportedSource Authentification
SQL Server / OracleSybase, Teradata, DB2, MySQL Windows, Basic
Files (CSV, XML, Text, Excel, Access) Folders Windows
SharePoint List (Online)SharePoint List (On-prem)
AnonymousAnonymous, Windows
OData Feed Anonymous, Windows, BasicKey (Data Market)
Azure Marketplace Key (Data Market)
Azure HDInsightAzure Blob StorageAzure Table Storage
Key (Azure Account)
SQL AzurePostgreSQL Basic
Teradata, Sybase, DB2, MySQL Windows, Basic
SSAS Tabular (Live Query) Windows
+SSIS Data Feed
Data Sources supported
Details on data sources supported https://support.office.com/en-us/article/Supported-Data-Sources-an
d-Data-Types
Prequisites Beware of versions Beware of types Drivers installed https://support.office.com/en-us/article/Data-source-prerequisites-P
ower-Query
HOW DOES IT WORK
Cloud On-Prem
Data Refresh explained
Power BIOnPrem Sources
DMG
Azure Service Bus
0Register DMGRegister Data Sources
1Ask for Data RefreshCheck permissionsFind Gateway
4 Execute Query
5 Get Data
6 Compress, Chunk
Queuerequest2
3 Readrequest
GatewayCloud Service
7 Update Model
Cloud On-Prem
OData access explained
Power BIOnPrem Sources
DMG
0 Index Sources
2Resolve Data SourceRedirect to local Gateway
4 Execute Query
3 Request Data
1 Search Source
Excel
Power Query
Hybrid Proxy
demoDMG IN ACTION
Data Source Discovery
Data Sources are declared on « Admin Center » They are associated to a Gateway
Gateway Cloud Service resolves Data Source and find right Gateway
Resolution = Provider + Server + Catalog Warning: Case sensitive
demoDATA SOURCE DISCOVERY
SECURITY
Excel
Power Query
Power BI
DMG
Data sources• SQL (1433)• FileShare• Etc.+DNS resolution
Azure Service Bus (outbound)• 9350-9354 / 5671,5672Power BI Admin Center (outbound)• 80/443
Direct HTTP (inbound)• HTTPS (8050)• HTTP (8051)
HTTPS (443)
Cloud
On-Prem
Network Security
Credential Manager
Identity Management
2 flavors : Impersonation by DMG Impersonation by Data Source
How is query authentified on Data Source ?
Impersonation by DMG
Connection identity is impersonated by DMG Peer-to-peer connection
Credentials are stored by DMG for each source Credentials can be stored encrypted locally Credentials can be stored encrypted on Cloud
Credentials are set up locally Credential Manager is a Click Once application Credential Manager discuss with DMG
demoCREDENTIAL MANAGER
Impersonation by Data Source
Used by Live Query Keep identity of original user Use the EffectiveUsername method
DMG authenticates to SSAS DMG passes EffectiveUsername property SSAS executes the query as if it was original
user DMG must be SSAS Admin
demoEffectiveUsername
HIGH AVAILABILITY &DISASTER RECOVERY
As easy as scale-out
DMGs
Cloud
On-Prem
Power BI
…Instance 1 Instance 2
Server A Server B
HA & DR
Gateways and Data sources are managed by Cloud
Credentials are encrypted by MDG Use of Certificate, self-generated or yours Need Certificate when registering a new instance Warning when stored localy
Load Balancing
demoHA & DR
Going further
Express Route Mashup Container (aka Power Query) Monitoring & Logging
Resources
Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help-5e391ecb-500c-47a3-bd0f-a6173b541044
Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx Supported Data sources :
https://support.office.com/en-us/article/Supported-Data-Sources-and-Data-Types DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931 How does DMG work - sequence diagram : https://
support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway-078D306D-2D28-409C-962B-D80B69A1D0B2
Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653 Store encrypted credentials in the cloud for business continuity : https://
support.office.com/en-us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886-da51b8cadc5b
Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint-Conference/2014/SPC397
Mini Nay & Luming Han Session at PASS Summit : http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231
@djeepy1 - http://blog.djeepy1.net
Any questions ?
Thank You!