bluesoft @ aws re:invent 2017 + aws 101

AWS re:INVENT 2017@andrefaria

6º ReinventA Bluesoft esteve presente em todos eles!

13.000 Sessions

43.000+ attendees

Amazon 101

Antes do Cloud

Cloud Computing is on-demand delivery of compute power, database storage, applications and other IT resources

through a cloud services platform via the Internet with pay-as-you-go pricing.

no upfront investments no hardware management

low cost scalable

capex vs opex no capacity guessing

increased speed and agility focus on core activities

IaaS - Infrastructurenetworking, computers, storage (EC2)

PaaS - Platformmanagement layer, patching (RDS)

SaaS - Softwareend user apps (Amazon WorkDocs)

Global Infrastructuremore than 1M customers in 190 countries

low latency and higher throughput42 AZs - 16 regions

AWS Management ConsoleSimple and Intuitive User Interface

AWS Command Line Interface (CLI) AWS SDKs

EC2 - Amazon Elastic Compute Cloud

On-Demandpay by the hour

Reservedup to 75% for upfront payment

Spot bid on spare capacity

ECS - Ec2 Container Service

ECSContainer ManagementRun Containers Clusters on EC2 instances

ECR Container RegistryStore, Manage, and Deploy Containers

Compute Services

AWS Batchplan, schedule, and run batch computing jobs on AWS - it automatically provision resources (cpu, memory, spot, etc.) no need to manage servers clusters to run your jobs

Amazon Lightsaillaugh virtual private servers

Compute Services

AWS Lambdarun code without provisioning or maintaining servers, pay only for compute time you consume

Elastic Beanstalkrun Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Tomcat, Passenger, and Internet Information Services (IIS).

Compute and Storage Services

Amazon S3 and GlacierS3 is an object storage designed to have 99.999999999% durability

Glacier is a extremely low cost storage for archiving and long-term backup.

AWS AutoscalingEnsure that your are running the desired number of EC2 Instances, and increases instances if demand increases

Storage Services

Elastic File SystemSimple scalable file storage to use with EC2. Storage is elastic and can grow or shrink automatically as you add and remove files.

AWS EBSPersistent block storage volumes to use with EC2, automatically replicated within the AZ. Best for low latency storage.

Storage and Database Services

AuroraMySQL and Postgres compatible relational database that combines speed an availability of high-end comercial databases for 1/10 of the cost.

AWS Storage GatewayHybrid Storage between on premisses and cloud

Database Services

Amazon Dynamo DBfast and flexible NoSQL DB for consistent, single-digit millisecond latency at any scale, support both document and key-value storage.

Amazon RDSManaged Aurora, PostgreSQL, MySQL, MariaDB, Oracle, Microsoft SQL Server.

Database and Networking Services

Amazon VPCprovision a logically isolated section for the cloud to launch resources (ip ranges, subnets, routes, route tables).

Amazon ElastiCacheManaged InMemory Database. Supports Redis and Memcached.

Networking Services

Route 53Highly available and scalable DNS Web Service - translates domains in IPS addresses

AWS Cloud FrontGlobal Content Delivery Network (CDN) that accelerates delivery for websites, APIs, video, or other assets, routing automatically to the nearest edge location.

Networking and Developer Services

AWS Code CommitFully Managed Source Control Service to host private Git Repos.

Elastic Load BalancingELB automatically distributes incoming application traffic across multiple EC2 instances enabling fault-tolerance and scaling.

Developer Services

AWS Code DeployAutomates code deployments to any instance

AWS Code BuildFully Managed Build Services that compile source code, run tests and produces software packages that ready to deploy

Developer Services

AWS X-Rayanalyse and debug apps with end-to-end view of requests and a map of components.

AWS Code PipelineContinuous Integration and Continuous Delivery

Management Tools

AWS Systems Managercollect inventory, apply patches, create images, configure and run commands.

Amazon CloudWatchMonitoring Resources and Apps

Management ToolsAWS Cloud Trailrecords API calls for your account and delivers log files

AWS Cloud FormationCreate and manage a collection of AWS resources, providing and updating

Management ToolsAWS OpsWorksConfiguration Management Service that uses Chef or Puppet to automate how servers are configured, deposed and manger across EC2 instances.

AWS ConfigFull Managed service that provides an AWS resource inventory, config history, config change notifications and rules evaluations

Management ToolsAWS Services CatalogCreate and manage catalogs of IT services that are approved for use on AWS

AWS ConfigFull Managed service that provides an AWS resource inventory, config history, config change notifications and rules evaluations

Management ToolsAmazon InspectorAmazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices.

AWS IAMcontrol access to AWS services and resources for your users

Management and Analytics ToolsAmazon Athenais an serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. 94 Athena is serverless. You pay only for the queries that you run.

AWS IAMcontrol access to AWS services and resources for your users

Analytics ToolsAmazon CloudSearchmanaged search solution for websites and applications

Amazon Elasticsearchmanaged Elasticsearch

Amazon EMRmanaged service to run Hadoop, Spark, HBase, Presto, and Flink workloads that in a easy, fast, and cost-effective fashion to process vast amounts of data across dynamically scalable Amazon EC2 instances.

KinesisKinesis Firehosecapture, transforms and load streaming data into s3, redshift, kinesis analytics for real time analytics

Kinesis Analyticsprocess streaming data in real time with standard SQL without having to learn new languages or processing frameworks - run queries continuously

Kinesisplatform for collecting, storing and analysing streaming data - you can load terrabytes of data per hour from IoT devices, mobile apps, etc. Kinesis offers 3 services.

Kinesis Streamscontinuously capture and store treats of data per hour from thousands for sources

AnalyticsAmazon QuickSightcloud business analytics service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights.

Amazon Redshiftfast, fully managed, petabyte - scale data warehouse that makes it simple and cost - effective to analyze all your data using your existing business intelligence tools.

ETLAWS Gluefully managed ELT service that makes it easy to move data between data stores.

Disponibiliza um serviço ETL gerenciado, executado em um ambiente Apache Spark sem servidor.

Para não Spark, Hive, Pig, etc. usar Data Pipeline

Amazon Data PipelineMove data between different AWS compute and storage services. Access, Transform and Process at Scale and Store Results.

Serviço de orquestração com flexibilidade de ambiente de execução, do acesso e do controle sobre os recursos que executam código, bem como sobre o próprio código responsável pelo processamento dos dados.

IAAmazon Pollyturns text into Speech

Amazon Rekognitionimage analysis

Amazon LEXbuilding conversational interfaces into any application using voice and text

IA and MobileAWS Mobile Hubquickly create and configure mobile app backends features and integrate them to the mobile app

Amazon Cognitoserverless identity service

Amazon Pinpointtarget campaigns to drive user engagement (e-mails, SMS, push notifications)

Machine Learningmakes it easy for developers of all skill levels to use machine learning technology. Provides visualisation tools and wizards that guide you through the process of creating machine learn ing models

Amazon Device Farmtest apps on many devices at once (Android, iOS and web)

Mobile and Application ServicesSimple Workflow Servicedevelopers build, run, and scale background jobs that have parallel or sequential steps (like Step functions but no visual and more control of your logic)

Amazon API Gatewaycreate, publish, maintain and monitor secure APIs at scale

Mobile Analyticsmeasure app usage and revenue

AWS Step Functionscoordinate componentes of distributed applications and micro services using visual workflows

Messaging and App StreamingAmazon Workspacesfully managed desktop computing service

Amazon AppStream 2.0store your app from AWS to any device running in a web browser

Amazon SQSmanaged queuing service

Amazon SNSpush notification service

Amazon SESsend e-mails

IoTAWS IoTconnect devices to AWS

AWS Greengrassrun local compute, lambda, messaging, caching, syncing for connected devices

AWS ioT Buttonprogramable button (Dash alike)

back to re:INVENT

first keynote

$18B+ revenue run rate

42% growth rate

AWS 44% of the public cloud marketshare (more than all other competitors combined)

Millions of Active Customers airbnb, slack, intercom, pinterest, sony, go pro, johson, pfizer, GE,

philps, siemens, netflix, disney, hbo, discovery, fox, kellogs, coca cola, samsung, LG

3000 government agencies 8000 academic institutions

Pace of Innovation

2016 1000+ Features 2017 1300+ Features

What builders really want?

6 songs

second keynote

Voice is next big disruptionit’s a natural interface

how you interact with people?

Security is everyone’s job

Encryption is not only faster and more efficient now. There’s really no excuse not to encrypt your data.

Dance like no one is watching Encrypt like everyone is

Trusted Advisor

Inspector (app level)

GuardDuty (network level plus IA)

Serverless

Product LaunchAmazon Elastic Container Service for Kubernetes (EKS)

Go and .NET Support

Product LaunchAWS Serverless Application Repository

Publish and Find Lambdas to use

AI

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Vision Speech LanguageServices

Platforms

Frameworks

Infrastructure

Amazon ML Spark & EMR KinesisMechanical

TurkAmazon ECSAmazon Batch

AWS Deep Learning AMI

ApacheMXNet TensorFlow Caffe2

& Caffe Theano Keras CognitiveToolkit

PyTorch

GPU CPU IoT(Greengrass)

Mobile

Gluon

AWS DeepLensA camera fully loaded with onboard compute power optimized for deep learning

Amazon Sagemaker- prebuilt notebooks that solve common problems in machine learning - 10 algorithms to address problems - Import your own if you need a custom solution - “one-click training” specify the location of your dataset in S3, choose

an instance type to run the computation, and Sagemakers does all the heavy lifting, setting up the algorithms to run your training.

- “one-click-deploy“ set the instance type and minimum/maximum numbers for your cluster and Sagemaker then gives you secure endpoints to connect to your app.

Amazon Rekognition


Amazon Rekognit ion

Extract rich metadata from visual content

Object and SceneDetection

FacialAnalysis

FaceComparison

FacialRecognition

Celebrityrecognition

Imagemoderation

Text in Image


Maple

Villa

BushesGrass

Tree

House

Window

Sky

Mountain Range

Forest

Clouds

Object and scene detection makes it easy for you to add features that search, filter, and curate large image libraries.

Identify objects and scenes and provide confidence scores

DetectLabelsObject & Scene Detect ion


Demographic Data

Facial Landmarks

Sentiment Expressed

Image Quality

Brightness: 23.6Sharpness: 99.9

General Attributes

Fac ia l Analys is DetectFaces

Analyze facial characteristics in multiple dimensions

Smiling

99.1%

Female

100%

Mouth Closed

99.5%

Age Range

26 – 43 years old

Crowd Mode


Facia l Analys is

Image Quality

Facial Landmarks

Demographic Data Emotion Expressed

General Attributes

Facial PoseBrightness 23.6%Sharpness 99.9%

EyeLeft,EyeRight,NoseRightPupil,LeftPupil

MouthRight,LeftEyeBrowUpBounding Box...

Age Range 29-45Gender:Male 96.5%

Happy 83.8%Surprised 0.65%

Smile:True 23.6%EyesOpen:True 99.8%Beard:True 99.5%Mustache:True 99.9%...

Pitch 1.446Roll 5.725Yaw 4.383

DetectFaces

AWS Rekognition Videoprocess real-time and batch video to detect objects, people, activities, and more

- detect inappropriate content - check surveillance footage for missing people - continually trained, gets “smarter” as more people use it

Amazon Kinesis Video Streamsits real time streaming capabilities for video

- integrates with Rekognition video (as an input source) - SDK that manufacturers can use to integrate it directly into their devices


http://amzn.to/takeselfie


End-to-End ArchitectureSELECT

STREAM COUNT(*) AS MUSTACH_COUNT,STEP(ROWTIME BY INTERVAL '1' SECOND)

FROM SOURCE_STREAMWHERE HAS_MUSTACH = TRUE;

Amazon Kinesis Stream

Amazon Kinesis

Analytics

Amazon Cognito

Amazon Kinesis Stream

AmazonDynamoDB

Amazon Lambda

Amazon S3

JavaScript SDK

AmazonRekognition

Amazon Kinesis

Firehose

Amazon S3


Amazon Rekognit ion Case StudyW a s h i n g t o n C o u n t y S h e r i f f ’ s O f f i c e

Chris AdzimaWashington County Sheriff’s [email protected]


Real-wor ld example


The solut ion


Real-wor ld example


Unlimitedreplays

Returns an MP3 or audio stream

Lightning-fast response

Fully managed and low cost

Amazon PollyTurn text into lifelike speech using deep learning technologies to synthesize speech that sounds like a human voice

Potential use cases

Content creation Education and E-learning

Mobile and desktop applications Customer contact center

Internet of Things (IoT) Accessibility


Amazon LexConversational interfaces for your applications, powered by the same natural language understanding (NLU) and automatic speech recognition (ASR) models as Alexa

Integrated development in

AWS console

Trigger AWS Lambda

functions

Multi-step conversations

Continually improving ASR and

NLU models

Enterprise connectors

Fully managed

Potential use cases

Appointment booking Customer support (Contact Center bots)

Informational services Access enterprise data

Internet of Things (IoT)

Amazon Transcribeconverts speech to text

Amazon Translateit translates text from one language to another

use batches of text from S3 it boasts real-time translation

Amazon Comprehendfully managed natural language processing service

Provide data from your lake (S3) via an API then Comprehend will provide four elements for analysis:

1 Entities – Things like people, dates, and specific places 2 Key phrases – Comprehend picks out the “most important” sets of words 3 Language – Automatic detection of the language used 4 Sentiment – Is the text saying something positive or negative?

Amazon Comprehend

Customer Voice

Content Recommendation


The Alexa Family


“Book a hotel”Book hotel

NYC

“Book a hotel in NYC”

Automatic speech recognition

Hotel booking

New York City

Natural language understanding

Intent/slotmodel

UtterancesHotel bookingCity New York CityCheck In April 19Check Out April 21

“Your hotel is booked for April 19”

Amazon Polly Confirmation: “Your hotel is booked for April 19”

“Can I go aheadwith the

booking?”

a

in


Amazon Lex

UtterancesSpoken or typed phrases that invoke your intent

BookHotelIntentsAn intent performs an action in response to natural language user input

SlotsSlots are input data required to fulfill the intent

FulfillmentFulfillment mechanism for your intent


Amazon Lex—create a bot

1

2

Define sample utterances

Define slots

Create bot


2

3

Confirm transaction

Fulfill transaction

1 Elicitinformation

Interact with bot

Amazon Lex—interact with a bot

Alexa for BusinessAlexa for Business is a fully managed service for

Alexa voice-controlled devices at work.

just say, “Alexa, start the meeting.”

Mobile


MOBILE INDUSTRY TRENDS

Time spent in apps and growing

1.6T hours

Source: AppAnnie

New enterprise apps built with web technologies

50%+

Source: AWS

JavaScript – most commonly used programming

language

66.7%▲

Source: Stack Overflow


THREE SIMPLE STEPS1. Pick a Platform

Mobile Hub

2. Set Up Cloud Services

Native SDKs

3. Connect Your App

Mobile CLI AWS Amplify


MOBILE CLI : SETTING UP CLOUD SERVICES

> awsmobile init

> awsmobile user-signin enable

Initialize your app

Enable User Sign-in

Supported services:

• user-signin (Amazon Cognito)

• analytics (Amazon Pinpoint)

• database (Amazon DynamoDB)

• user-files (Amazon S3)

• cloud-api (API GW & AWS Lambda)

Web app deployment support:

• hosting (Amazon S3 and Amazon CloudFront)


APP DATA: HARD PROBLEMS REMAIN

Data requirements vary across devices and become harder

when multiple users share data

Users want instant access to data

Building scalable data-driven apps without learning

distributed systems concepts is hard

Users want to continue using their apps even with low or no connectivity


AWS AppSync


START BUILDING FAST


EASY ACCESS TO RICH DATA

{"id": "1","name": "Get Milk","priority": "1"

},"id": "2","name": "Go to gym","priority": "5"

},…

type Query {getTodos: [Todo]

}

type Todo {id: ID!name: Stringdescription: Stringpriority: Intduedate: String

}

query {getTodos {

idnamepriority

}}

Model data with application schema

Client requests what it needs

Only that data is returned


INTEGRATE WITH YOUR APPLICATION

AWS AmplifyAWS Amplify is a JavaScript library for frontend and mobile

developers building cloud-enabled applications.

The library is a declarative interface across different categories of operations in order to make common tasks easier to add into

your application.

"I was able to use AWS Cognito to integrate a full-featured authentication system, including email signup verification and

MFA, into a React application with less than 10 lines of code, in about 20 minutes. That time would have been drastically less if I

had already created a user pool beforehand."

Networking


Load balancer used to route incoming requests to multiple Amazon EC2 instances, containers, or IP addresses in your VPC.

ELB

EC2Instance

EC2Instance

EC2Instance


Application Load Balancer Network Load Balancer Classic Load Balancer

Protocol HTTP, HTTPS, HTTP/2 TCP TCP, SSL, HTTP, HTTPS

SSL offloading ✔ ✔IP as Target ✔ ✔Path-based routing, Host-based routing ✔Static IP ✔WebSockets ✔ ✔Container Support ✔ ✔


For TCP in VPC, use Network Load Balancer.

For all other use cases in VPC , use Application Load Balancer

For Classic networking, use Classic Load Balancer

Aurora Multi-Master

Aurora ServerlessDatabase starts up on demand, shuts down when it’s not in use…

when it is in use, you’re billed by the second.

DynamoDB Global TablesThe first fully managed, multi-master, multi-region

database system in the world

DynamoDB On-Demand Backupcreate backups (and restore them) with one click or API call

Amazon NeptuneNeptune is a fully managed graph database service

store billions of records will millisecond latency

S3 Select and Glacier SelectGood for Data Lakes

pull out only the data you need using an API to pull only specific parts of an object


Amazon S3 By The Numbers

44 Availability Zones(16 more coming in 2018)

16 Regions(5 more coming in 2018)

Trillions of objects

Millions of requests per second

One of first three AWS Services

(2006)

99.999999999% Durability


Amazon S3 Avai lab i l i ty Zones

S3 stores data in at least 3 Availability Zones (AZ’s)

Each AZ can be up to 8 physical data centers

Unavailability of a data center or an AZ does not impact

overall S3 availability

Low latency private network connect data

centers and AZ’s

Physically separate – even extremely uncommon disasters would only affect a single AZ

Data is automatically distributed across a minimum of 3 AZ’s GEO separated within an AWS Region


Amazon S3 Storage Classes& T r a n s i t i o n s

S3 Standard S3 Standard –Infrequent Access

Amazon Glacier

Active data

Synchronous access

Milliseconds retrieval

2.1¢-GB/mo

Archive data

Asynchronous access

Minutes-to-hours retrieval

0.4¢-GB/mo

Infrequently accessed data

Synchronous access

Milliseconds retrieval

1.25¢-GB/mo


Amazon S3 Secur i ty , Enc rypt ion & Compl ianceT h e b r o a d e s t s e t o f t o o l s i n t h e i n d u s t r y

Security• IAM and Bucket Policies• Access Control Lists• Audit logging with CloudTrail

& Alerts with CloudWatch• Secure CloudFormation

templates• Amazon Macie• S3 Console Permission Checks

Encryption• Encryption in transit with TLS• SSE-S3 – Amazon S3 manages

data & keys• SSE-C – Customer managed keys• SSE-KMS – Master keys in KMS• CSE – 100% Customer managed• Default Bucket Encryption• Encryption Status in Inventory

Compliance• PCI-DSS• HIPAA/HITECH• FedRAMP• FISMA• EU Data Protection

Directive


Do More With Your In -place Data

• Athena• Redshift Spectrum• QuickSight• EMR

Data Lake Storage IoT Storage Machine Learning

& AI Storage

• AWS IoT• Greengrass• Other IoT sensors

• Rekognition• LEX• Polly• MXNet & TensorFlow


Enter Data Lake Architectures

Data lake is a new and increasingly popular architecture to store and analyze massive volumes and heterogeneous types of data


Benefits of a Data Lake—All Data is in One Place

Analyze all of your data, from all of your sources, in one stored

location

“Why is the data distributed in many locations? Where is the

single source of truth?”


Designed for 11 9s of durability

Designed for 99.99% availability

Durable Available High performance� Multiple upload� Range GET� Scalable throughput

� Store as much as you need� Scale storage and compute

independently� No minimum usage commitments

Scalable� Amazon EMR� Amazon Redshift Spectrum� Amazon DynamoDB� Amazon Athena� AWS Glue� Amazon Rekognition� Amazon Macie

Integrated� Simple REST API� AWS SDKs� Simple management tools� Event notification� Lifecycle policies

Easy to use

Why Amazon S3 for a Data Lake?


AWS Direct Connect AWS Snowball ISV Connectors

Amazon Kinesis Firehose

Amazon S3 TransferAcceleration

AWS StorageGateway

Data Ingestion into Amazon S3


Building a Data Lake on AWS

Kinesis FirehoseAthena

Query Service


Processing and Analytics

Real-time Batch

AI and Predictive

BI and Data Visualization

Transactional and RDBMS

AWS LambdaApache Storm

on EMR

Apache Flink on EMR

Spark Streaming on EMR

Elasticsearch Service

Kinesis Analytics, Kinesis Streams

DynamoDB

NoSQL DB Relational DatabaseAurora

EMRHadoop, Spark,

Presto

Amazon RedshiftData Warehouse

Amazon AthenaQuery Service

Amazon LexSpeech recognition

Amazon Rekognition

Amazon PollyText to speech

Machine LearningPredictive analytics

Kinesis Streams and Firehose

IoT is about…“closing the gap between the physical and digital

world in self-reinforcing and self-improving systems.”

AWS IoT 1-Click

AWS IoT Device ManagementIoT Device Management is similar to 1-Click, but at a larger scale.

Onboard, deploy, and manage your fleet of devices all from a single location. Organize inventory, query the fleet for troubleshooting, and remotely deploy updates

take action on subsets of your devices, not just all of them at once

AWS IoT Device DefenderMany of the attacks we’ve seen in recent years have

utilized unsecured IoT devices.

Device Defender allows you to set device policies, audit them, and monitor behaviours on an individual level to identify anomalies and out-of-compliance behaviours

Send you automatic alerts when it detects a problem

AWS IoT AnalyticsTraditionally, IoT devices pick up a lot of “noisy” data, like

temperature and humidity, resulting in raw, unstructured information that’s very difficult to process.

AWS FreeRTOSWhile larger devices often come with a full onboard CPU, smaller ones tend

to use an MCU (micro controller unit) and they do still need an operating system. Amazon has created their own version of FreeRTOS (a commonly

used OS in these devices), and it’s got some awesome features.

Amazon FreeRTOS comes with prepackaged libraries to connect to AWS services, update, and secure the device. It also allows you to easily send

data to the cloud for further analysis.

AWS GreenGrassExecute funções lambda nos dispositivos com segurança usando recursos locais de computação, sistema de mensagens, armazenamento de dados

em cache e sincronização para dispositivos conectados.

AWS GreenGrass Exemplo

You have a collection of IoT sensors deployed in the field, along with a GreenGrass device. Rather than sending data straight to the cloud, the

sensors can connect to the GreenGrass device directly and have it perform some operation for them. This is done locally – the sensors don’t need to be

connected to the public internet to communicate with GreenGrass.

This saves time by lowering the latency of connections, and money by filtering data from the sensors before sending it all to the cloud for

processing.

AWS GreenGrass Machine Learning Inference

GreenGrass device still operates the same way – at the edge of your network. But it can now apply machine learning models in the field.

ex: an IoT sensor that takes an action in response to a voice command. Before, you’d have to send that data to the cloud for processing, then back to the sensor, which would then trigger the action (back to the cloud again, most likely). With Machine Learning Inference on a GreenGrass device, you can do

all of that locally, resulting in much faster response times.

Obrigado!@andrefaria - andrefaria.com

http://andrefaria.com

bluesoft @ aws re:invent 2017 + aws 101

Technology