azure data platform overview

Azure Data Overview…dove metto i miei dati?...

Agenda

Azure SQL Database

Azure DocumentDB

Azure Table Storage

Azure Redis Cache Azure Search

…almeno fino a ieri

Agenda (post //build/)

Azure SQL Database

Azure CosmosDB(DocumentDB, MongoDB, Graph, Key-value)

Azure Table Storage

Azure Redis Cache Azure Search

Azure Database for MySQL

Azure Databasefor PostgreSQL

Azure SQL Database

SQL Server (everywhere)

Azure SQL Database

Azure SQL Database is a relational database-as-a-service (DBaaS) hosted in the Azure cloud that falls into the industry categories

of Software-as-a-Service (SaaS) and Platform-as-a-Service (PaaS).

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-paas-vs-sql-server-iaas

Azure SQL Database

Azure SQL Database is optimized to reduce overall costs to the minimum for provisioning and managing many databases. It reduces

ongoing administration costs because you do not have to manage any virtual machines, operating system or database software.

Azure SQL Database

Azure SQL Database is optimized to reduce overall costs to the minimum for provisioning and managing many databases. It reduces

ongoing administration costs because you do not have to manage any virtual machines, operating system or database software.

You do not have to manage:

Azure SQL Database

Azure SQL Database is optimized to reduce overall costs to the minimum for provisioning and managing many databases. It reduces

ongoing administration costs because you do not have to manage any virtual machines, operating system or database software.

You do not have to manage:

upgrades

Azure SQL Database

Azure SQL Database is optimized to reduce overall costs to the minimum for provisioning and managing many databases. It reduces

ongoing administration costs because you do not have to manage any virtual machines, operating system or database software.

You do not have to manage:

high availability

Azure SQL Database

Azure SQL Database is optimized to reduce overall costs to the minimum for provisioning and managing many databases. It reduces

ongoing administration costs because you do not have to manage any virtual machines, operating system or database software.

You do not have to manage:

backups

Service tiers

• Basic: Best suited for a small database, supporting typically one single active operation at a given time. Examples include databases used for development or testing, or small-scale infrequently used applications.

• Standard: The go-to option for cloud applications with low to medium IO performance requirements, supporting multiple concurrent queries. Examples include workgroup or web applications.

• Premium: Designed for high transactional volume with high IO performance requirements, supporting many concurrent users. Examples are databases supporting mission critical applications.

• Premium RS: Designed for IO-intensive workloads that do not require the highest availability guarantees. Examples include testing high-performance workloads, or an analytical workload where the database is not the system of record.

Service tiers

• After initially picking a service tier and performance level, you can scale a single database up or down dynamically

• Changing the service tier and/or performance level of a database creates a replica of the original database at the new performance level, and then switches connections over to the replica.

• No data is lost during this process but during the brief moment when we switch over to the replica, connections to the database are disabled, so some transactions in flight may be rolled back.

Create Azure Sql Database

• Azure Portal

• Azure CLI

• Powershell

Demo

Elastic pools

• SQL Database elastic pools are a simple, cost-effective solution for managing and scaling multiple databases that have varying and unpredictable usage demands

• Within the pool, individual databases are given the flexibility to auto-scale within set parameters. Under heavy load, a database can consume more eDTUs to meet demand. Databases under light loads consume less, and databases under no load consume no eDTUs.

Elastic pool

Elastic pool

Elastic pool

Elastic pool

Elastic pool

Demo

Business continuity

• Backup

• Active Geo-Replication

Business continuity: backup

• SQL Database automatically performs a combination of full database backups weekly, differential database backups hourly, and transaction log backups every five minutes to protect your business from data loss. These backups are stored in geo-redundant storage

• Use automated backups as your business continuity and recovery mechanism if your application:• Is not considered mission critical

• Has a low rate of data change (low transactions per hour) and losing up to an hour of change is an acceptable data loss

• Is cost sensitive

Business continuity: active geo-replication

• Is possible to use Active Geo-Replication to configure a database to have up to four readable secondary databases in the regions of your choice

• These secondary databases are kept synchronized with the primary database using an asynchronous replication mechanism

• Active Geo-Replication can also be used to provide better query performance for read-only queries to geographically dispersed users.

Demo

Monitoring

• Metrics

• SQL Database advisor

• Query Performance Insight

Demo

Azure Database Migration Service

• Using database migration service simplifies the migration of existing on-premises SQL Server, Oracle, and MySQL databases to Azure, whether your target database is Azure SQL Database, Azure SQL Database Managed Instance or Microsoft SQL Server in an Azure virtual machine.

Azure Database Migration Service

• Using database migration service simplifies the migration of existing on-premises SQL Server, Oracle, and MySQL databases to Azure, whether your target database is Azure SQL Database, Azure SQL Database Managed Instance or Microsoft SQL Server in an Azure virtual machine.

PostgreSQL / MySQL

Azure database for PostgreSQL/MySQL

• Azure Database for PostgreSQL/MySQL is a relational database service in the Microsoft cloud built for developers based on the community version of open source PostgreSQL and MySQL Community Edition database engine. It provides:• Predictable performance at multiple service levels

• Dynamic scalability with no application downtime

• Built-in high availability

• Data protection

Demo

Azure Table Storage

Azure Table Storage

Azure Table storage is a service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design

https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-how-to-use-tables

Table service concepts

• Storage Account

• Table: a table is a collection of entities

• Entitiy: An entity is a set of properties (similar to a database row). An entity can be up to 1MB in size

• Properties: A property is a name-value pair. Each entity can include up to 252 properties to store data

Table entities

• Table entities represent the units of data stored in a table and are similar to rows in a typical relational database table

• Each entity defines a collection of properties

• Entities must define the following three system properties as part of the property collection:• PartitionKey

• RowKey

• Timestamp

Table partitions

Queries performance

Azure Storage: paper

http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf

Azure CosmosDB

Azure CosmosDB

Azure Cosmos DB is Microsoft's globally distributed,

multi-model database

https://docs.microsoft.com/en-us/azure/cosmos-db/introduction

Azure CosmosDB

Global distribution

• Within a given region, all resources are horizontally partitioned using resource partitions (local distribution).

• Each resource partition is also replicated across geographical regions (global distribution).

Multi-model, multi API-support

• The core content-model of Cosmos DB’s database engine is based on atom-record-sequence (ARS)• Atoms consist of a small set of primitive

types like string, bool, and number. Records are structs composed of these types. Sequences are arrays consisting of atoms, records, or sequences.

• The database engine can efficiently translate and project different data models onto the ARS-based data model.

Multiple consistency models

• It’s possible to configure the default consistency level on Cosmos DB account, and later override the consistency on a specific read request

Guaranteed SLA

• Availability: 99.99% uptime availability SLA for each of the data and control plane operations.

• Throughput: 99.99% of requests complete successfully

• Latency: 99.99% of <10 ms latencies at the 99th percentile

• Consistency: 100% of read requests will meet the consistency guarantee for the consistency level requested by you.

Demo

CosmosDB: indexing

• CosmosDB indexes, by default, each document’s attribute

• Automatic indexing of documents without requiring schema or secondary indexes is a key capability of CosmosDB and is enabled by write-optimized, lock-free and log-structured index maintenance techniques

CosmosDB: indexing policy

• The indexing policy of each collection allows you to make performance and storage trade-offs associated with indexing• Choose whether the collection automatically indexes all of the documents or

not

• Choose whether to include or exclude specific paths or patterns in your documents from the index

• Choose between synchronous (consistent) and asynchronous (lazy) index updates

• The indexing policy can be changed

Demo

Azure Redis Cache

Azure Redis Cache

• Redis is an advanced key-value store, where keys can contain data structures such as strings, hashes, lists, sets, and sorted sets. Redissupports a set of atomic operations on these data types.

• Azure Redis Cache uses Redis authentication and also supports SSL connections to Redis.

Service tiers

• Basic: Single node, multiple sizes, ideal for development/test and non-critical workloads. The Basic tier has no SLA.

• Standard: A replicated cache in a two-node primary/secondary configuration managed by Microsoft, with a high-availability SLA.

• Premium: All of the Standard tier features, plus:

• Redis data persistence: allows you to persist the cache data in an Azure Storage account

• Redis cluster

• Enhanced security and isolation

• Reboot

• Schedule updates

Demo

Azure Search

Azure Search

Azure Search is a cloud search-as-a-service solution that delegates server and infrastructure management to Microsoft, leaving you with a ready-to-use service that you can populate with your data and then use

to add search to your web or mobile application.

https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search

Azure Search

• Full text search and text analysis• Search suggestions

• Facet queries

• Scoring

• …

• Language support

• Data integration

Service tiers

• Free: A shared service, at no charge, used for evaluation, investigation, or small workloads.

• Basic: Small production workloads on dedicated hardware. Highly available.

• Standard:• S1

• S2

• S3

• S3 HD

Indexers

• An indexer in Azure Search is a crawler that extracts searchable data and metadata from an external data source and populates an index based on field-to-field mappings between the index and your data source.

• You can use an indexer as the sole means for data ingestion, or use a combination of techniques

• You can run indexers on demand or on a recurring data refresh schedule

Indexer types

• Azure Blob Storage indexer

• Azure Table Storage indexer

• Azure SQL indexer

• Azure CosmosDB indexer

Demo

Wrap-up

SQL IaaS vs SQL Database• Existing applications that

require fast migration to the cloud with minimal changes.

• SQL Server instances with up to 64 TB of storage. The instance can support as many databases as needed.

• Migrating and building enterprise and hybrid applications.

• New cloud-designed applications that have time constraints in development and marketing.

• Databases of up to 1 TB, or larger databases that can be horizontally or vertically partitioned using a scale-out pattern.

• Building Software-as-a-Service (SaaS) applications.

SQL IaaS vs SQL Database: features

• https://docs.microsoft.com/en-us/azure/sql-database/sql-database-features

Extra-time

Transient fault handling

Transient faults can occur in any environment, on any platform or operating system, and in any kind of application

Transient fault handling: cloud

• Many resources in a cloud environment are shared, and access to these resources is subject to throttling in order to protect the resource

• Cloud environments are built using vast numbers of commodity hardware units

• There are often more hardware components, including network infrastructure such as routers and load balancers, between the application and the resources and services it uses

• Network conditions between the client and the server may be variable

Transient fault handling: challanges

• The application must be able to detect faults when they occur, and determine if these faults are likely to be transient, more long-lasting, or are terminal failures

• The application must be able to retry the operation if it determines that the fault is likely to be transient

• The application must use an appropriate strategy for the retries:• Exponential back-off

• Incremental intervals

• Regular intervals

• Immediate retry

• Randomization

About me

Alessandro Melchiori

@amelchiori

http://melkio.codiceplastico.com