![Page 1: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/1.jpg)
Principal Architect
Private Apps in the Public CloudIssac Goldstand
![Page 2: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/2.jpg)
IntroducingAppCloud
![Page 3: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/3.jpg)
Introducing AppCloud
Out-of-the-Box ExperienceUsers discover applications as they set up a new PC or smartphone
App Discovery EngineUsers browse for software in a curated catalog
![Page 4: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/4.jpg)
Introducing AppCloud
Dynamic NotificationsRe-engaging users when it makes sense
AnalyticsUnderstanding users.
![Page 5: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/5.jpg)
Behind the scenes...
Introducing AppCloud
Sponsored AppsPopular Apps
AppCloud Catalog
* Popular Apps - Free apps that users are likely to install on their device* Sponsored Apps - Apps with campaigns that can generate revenue
Mix of popular and sponsored
apps
App Personalization Engine
Sponsored App
![Page 6: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/6.jpg)
Developing the MVP
![Page 7: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/7.jpg)
Developing the MVP
![Page 8: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/8.jpg)
Developing the MVP
![Page 9: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/9.jpg)
Design Considerations
![Page 10: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/10.jpg)
2 Major Concerns
![Page 11: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/11.jpg)
“The more possessions one owns, the more worries one
needs to deal with”
“ , דאגות מרבה נכסים ”מרבה
![Page 12: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/12.jpg)
Things got...
Design Considerations
![Page 13: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/13.jpg)
Things got… ...pretty crazy
Design Considerations
![Page 14: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/14.jpg)
What happens if an attacker breaches one of the servers?
![Page 15: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/15.jpg)
Design Considerations
![Page 16: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/16.jpg)
Design Considerations
![Page 17: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/17.jpg)
We need to partition our environments properly
![Page 18: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/18.jpg)
Two types of environments
![Page 19: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/19.jpg)
Two types of environments
Single TenantMultiple Tenant
![Page 20: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/20.jpg)
Separate sensitive components
![Page 21: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/21.jpg)
In a modern cloud, where most hardware is multi-
tenant by definition, how can we accomplish single tenant
partitioning?
![Page 22: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/22.jpg)
Hardware (Logical)Network (Logical)
Application
![Page 23: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/23.jpg)
Hardware Layer
![Page 24: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/24.jpg)
TL; DR
We use dedicated compute instance per
component/environment (customer)
![Page 25: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/25.jpg)
Networking Layer
![Page 26: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/26.jpg)
Network
How we used to do it?
https://commons.wikimedia.org/wiki/File:3_men_working_on_a_portable_phone_switchboard.jpg
![Page 27: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/27.jpg)
Each customer gets their own (set of) VLAN(s)
![Page 28: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/28.jpg)
No interconnectivity between customer VLANs
![Page 29: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/29.jpg)
Additional VLAN(s) for shared components
![Page 30: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/30.jpg)
AWS
![Page 31: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/31.jpg)
AWSVLAN == VPC
![Page 32: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/32.jpg)
AWSVLAN == VPC
(loosely speaking)
![Page 33: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/33.jpg)
Network Layer
Split each customer to their own VPC
![Page 34: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/34.jpg)
AWS security groups
![Page 35: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/35.jpg)
Network Layer
AWS Jump Rules
Target is another (or the same) security group
![Page 36: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/36.jpg)
Network Layer
Each customer/component pair has a security group (at least one)
Allows fine-grained control of which services can access different sets of data
Note the separation of S3 buckets + use of IAM roles to access the S3 data
![Page 37: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/37.jpg)
Each customer/component pair has a security group (at
least one)
![Page 38: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/38.jpg)
Allows fine-grained control of which services can access
different sets of data
![Page 39: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/39.jpg)
In addition to security, S3 replication allows for cross-
region deployments
https://github.com/issacg/s3sync
![Page 40: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/40.jpg)
Application Layer
![Page 41: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/41.jpg)
Application LayerExample Workflow
1) Back-end sends app to “Publish App” microservice
2) “Publish” microservice stores data in S3 storage
3) “Publish” microservice calls “Parse App Metadata” and “Sign App” microservices
4) “Publish” microservice saves metadata + signature to database
Future slides will use PoV of “Publish App” microservice
![Page 42: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/42.jpg)
“Elastic applications in a public cloud should support
zero-configuration”
![Page 43: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/43.jpg)
“Elastic applications in a public cloud should support
zero-configuration” - Me
![Page 44: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/44.jpg)
Zero configuration allows us to support both auto-scaling groups and auto-healing in case of (many, but not all)
problems
![Page 45: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/45.jpg)
Zero-Configuration
NetworkingService Discovery
Credentials/Identity Management
Application + Config
![Page 46: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/46.jpg)
Networking
![Page 47: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/47.jpg)
Service Discovery
![Page 48: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/48.jpg)
Options
![Page 49: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/49.jpg)
Option 1
AWS Internal ELBRoute53 Private Hosted
Zones
![Page 50: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/50.jpg)
Service Discovery
Route53 Private Zone points to Internal ELB
ELB load balances traffic between Publish workers
If a publish worker fails the ELB health check, it is removed from the pool of healthy workers
![Page 51: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/51.jpg)
Option 2
Standalone service discovery
ZooKeeper, Consul, etc
![Page 52: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/52.jpg)
Consul
![Page 53: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/53.jpg)
Rich feature-set built-in
Service discoveryKV storage
Global mutex/semaphoresLeader electionHigh availability (active/active)
Encryption (Gossip + HTTP/RPC)
Health checks
![Page 54: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/54.jpg)
Incredibly elastic
Fits the cloud well
![Page 55: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/55.jpg)
Consul
Register current instance as “publish-i1234567890abcdefa.node.customerA” and “publish.service.customerA” + healthchecks
Addresses of well-known “parseapp” and “signapp” services via service discovery
Name of S3 bucket + path via KV storage
S3 access via IAM role
Database (host, user, password) via KV storage
![Page 56: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/56.jpg)
Zero-Configuration via Consul
Application LayerServer comes up via configuration management scripts
Server joins consul cluster
Server fetches application configuration based on well-known locations in the consul KV store
Server fetches application bits and boots
Service registers with consul (including healthcheck)
![Page 57: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/57.jpg)
Instead of looking up an ELB with a well-known hostname,
we can use a well-known service name and connect to
any machine inside that service group
![Page 58: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/58.jpg)
Service Discovery
Consul DNS lookup for “publish” service
Consul randomly picks a healthy instance and returns the address of the worker
If a publish worker fails the consul health check, it is removed from the pool of healthy workers
If we have a leader/follower app, we can use consul “tags” to get a specific instance (eg. master.publish.service…..consul)
![Page 59: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/59.jpg)
Application Layer
Service Discovery
redis.service.nyc3.consul / redis.service.consul
![Page 60: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/60.jpg)
Application Layer
https://www.flickr.com/photos/cogdog/566323330
Service Discovery
![Page 61: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/61.jpg)
Consul Architectural Concepts
![Page 62: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/62.jpg)
Application Layer
Consul High-level architecture
https://www.consul.io/docs/internals/architecture.html
![Page 63: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/63.jpg)
Two gossip pools - WAN & LAN
![Page 64: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/64.jpg)
LAN pools encapsulate a single (virtual) datacenter
Divided into server and client agents
![Page 65: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/65.jpg)
In each DC, a single server is elected as “Leader”
![Page 66: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/66.jpg)
Transactions are forwarded and committed to all servers
![Page 67: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/67.jpg)
Leader is responsible for maintaining consistency in
its DC
![Page 68: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/68.jpg)
WAN pool spans all datacenters (servers only)
![Page 69: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/69.jpg)
Cross-datacenter requests use RPC-forwarding
(between server nodes) to query the remote DC
![Page 70: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/70.jpg)
No DC stores information about other DCs
![Page 71: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/71.jpg)
Rich ACL systemWho can access what?
![Page 72: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/72.jpg)
Application Layer
http://imgur.com/gallery/WlgnC
Consul
![Page 73: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/73.jpg)
We already split everything into VPCs
![Page 74: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/74.jpg)
Each VPC becomes a DC in consul
![Page 75: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/75.jpg)
Each environment (customer) automatically
gets a private KV store and private service registry
![Page 76: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/76.jpg)
Shared services live in their own well-known dedicated DC with their own “shared” KV store & service registry
![Page 77: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/77.jpg)
Application Layer
https://www.flickr.com/photos/mherzber/500917537
Consul
![Page 78: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/78.jpg)
It’s possible to perform cross-datacenter queries
Controllable via ACLs
![Page 79: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/79.jpg)
Application Layer
Publish asks Consul Blue - who is local signapp? signapp.service.consul
Consul Blue answers with gossipped address of random signapp instance in healthy state
![Page 80: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/80.jpg)
Application Layer
Publish asks Consul Blue - who is tagged as leader node of parse in green DC? leader.parse.service.green.consul
Consul Blue checks the WAN gossipped peers for a server address of Consul Green cluster
Consul Blue forwards the query (via HTTP/S) to Consul Green over the WAN
Consul Green answers with gossipped address of a parse node with the tag “leader” and in healthy state
![Page 81: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/81.jpg)
Is that good enough?
Probably
![Page 82: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/82.jpg)
Application Layer
http://onceuponyourprime.com/2014/03/20/must-you-always-cross-your-eyes-and-dot-all-your-teas/
![Page 83: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/83.jpg)
Application Layer
http://monteeggers.com/shiny-object-syndrome-killing-business
![Page 84: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/84.jpg)
![Page 85: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/85.jpg)
Secure storage and audit control of private data
![Page 86: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/86.jpg)
One-time, short-lived, audited passwords
![Page 87: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/87.jpg)
Growing ecosystem of backends supporting one-
time-passwords
AWS-STS, MySQL, PostgreSQL, SSH, PKI,
Consul
![Page 88: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/88.jpg)
Application Layer
VaultHigh-level architecture
https://www.vaultproject.io/docs/internals/architecture.html
![Page 89: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/89.jpg)
Consul + Vault
Consul + Vault access via Vault (via Provisioning Service)
Addresses of well-known “parseapp” and “signapp” services via consul service discovery
Name of S3 bucket + path via KV storage (access via IAM Role*) * Could also use Vault AWS backend
Database host via consul KV storage
Database user, password via Vault
Register current instance for consul service discovery
![Page 90: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/90.jpg)
Zero-Configuration via Consul & Vault
NetworkingService DiscoveryCredentials/Identity ManagementApplication + Config
Application LayerServer comes up via configuration management scripts
Server identifies itself to Vault-backed provisioning service and gets consul SSL keypair + Consul access token + Vault access token for future queries
Server joins encrypted Consul cluster
Server fetches application bits and boots
Service fetches application configuration from Vault secret backend and Consul KV store
Service registers with consul (including healthcheck)
![Page 91: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/91.jpg)
The Challenge
![Page 92: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/92.jpg)
Application Layer
Provisioning Service
https://41.media.tumblr.com/eeb9825c9b3bf3a968d8ed63844b11df/tumblr_inline_nvrau6JwQD1rrhq52_540.jpg
![Page 93: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/93.jpg)
How do you bootstrap access for a single image running in
multiple instances (eg, an AMI in an auto-scaling
group)?
![Page 94: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/94.jpg)
We want to audit each machine’s access
individually - no shared authentication
![Page 95: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/95.jpg)
We don’t want to allow multiple machines (or
“anything”s) to authenticate the same token twice
![Page 96: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/96.jpg)
We don’t want to store secrets in a non-secret place
![Page 97: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/97.jpg)
Many suggestions for inclusion as an
authentication plugin for Vault
![Page 98: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/98.jpg)
I haven’t seen any I like
![Page 99: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/99.jpg)
Let Vault focus on protecting the data, extend it with
external tooling to fit your needs
![Page 100: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/100.jpg)
Provisioning Service
![Page 101: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/101.jpg)
Amazon EC2 Instance Identity Document
http://169.254.169.254/latest/dynamic/instance-identity/
![Page 102: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/102.jpg)
Includes embedded cryptographic signature to authenticate the document
![Page 103: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/103.jpg)
Signed by AWS
![Page 104: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/104.jpg)
Provisioning ServiceAWS EC2 Identity Document
Application LayerSubset of fields in the Identity Document
● AWS Account Number
● Instance ID
● Instance Primary Private IP address
● AMI + Kernel ID
● Launch request time
![Page 105: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/105.jpg)
Missing component role
![Page 106: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/106.jpg)
Missing environment (customer) identifier
![Page 107: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/107.jpg)
We currently store those in the EC2 user-data to be
processed by our configuration management
system
![Page 108: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/108.jpg)
After authenticating the instance the provisioning
service queries EC2 to obtain the user-data
It’s flexible to harden this later
![Page 109: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/109.jpg)
Provisioning ServiceInstance sends its identity document to Provisioning Service (PrvSrv)
PrvSrv authenticates the AWS signature
PrvSrv verifies that IP making the request matches the IP in the doc
PrvSrv verifies that the AWS account, AMI are whitelisted
PrvSrv uses the instance id to query the EC2 API to fetch additional metadata
Using this metadata, PrvSrv requests/generates credentials for Vault + Consul and returns this info to the instance
PrvSrv sends additional Vault token to bootstrap Consul
![Page 110: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/110.jpg)
EC2 Instance IDs are globally unique across accounts and
are never recycled
![Page 111: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/111.jpg)
Provisioning service will only provide a single token for an
instance
![Page 112: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/112.jpg)
Instances are guaranteed to be coming from inside our AWS accounts, and from a
verified IP address
![Page 113: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/113.jpg)
Consul bootstrap information
![Page 114: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/114.jpg)
TLS keys for the node (PKI backend)
https://github.com/issacg/vault-pki-client
![Page 115: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/115.jpg)
Current gossip shared-key (Generic backend)
![Page 116: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/116.jpg)
Token for consul (Consul backend)
![Page 117: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/117.jpg)
Provides vault token for application (no backend)
![Page 118: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/118.jpg)
No built-in Vault backend for Vault
![Page 119: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/119.jpg)
No built-in Vault backend for Vault
Not an unsolvable problem - provisioning service can take
care of this
![Page 120: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/120.jpg)
Future Plans & Challenges
![Page 121: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/121.jpg)
Separate Vault per environment (eg, Vault per
Consul DC)
![Page 122: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/122.jpg)
How do we manage unsealing with so many
Vault clusters?
![Page 123: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/123.jpg)
How do we pass the secrets from the provisioning service
client to the application service in a secure manner?
![Page 124: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/124.jpg)
How do we need to change the provisioning service to
run with containers?
![Page 125: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/125.jpg)
Summing Things Up
![Page 126: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/126.jpg)
In a modern cloud, where most hardware is multi-
tenant by definition, how can we accomplish single tenant
partitioning?
![Page 127: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/127.jpg)
PartitioningHardware (Logical) Layer
Networking (Logical) LayerApplication Layer
![Page 128: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/128.jpg)
Hardware (Logical) Layer
Instance per-service per-environment
![Page 129: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/129.jpg)
Networking (Logical) Layer
VPC per environment
Security Group per component/environment
compute-instance
![Page 130: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/130.jpg)
Application Layer
Separation of shared / private microservices
Consul + Vault + Provisioning service to
provide partitioned zero-configuration
![Page 131: Private Apps in the Public Cloud - DevConTLV March 2016](https://reader034.vdocuments.net/reader034/viewer/2022051706/58ec6b5d1a28abee528b4771/html5/thumbnails/131.jpg)
Questions?