automation for the humans
TRANSCRIPT
Automation for the HumansDeveloping automation with the humans in mind
Michael DucyRakuten Technology Conference - 22/10/2016
What do you have to think about immediately after coming up with the idea for an application?
Infrastructure
PRODUCTION?
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
PRODUCTION?
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
PRODUCTION?
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
PRODUCTION?
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
PRODUCTION?
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
$SUPERVISION
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
$MONITORING
$APP $DB$APP
$APP $DB$APP
$APP $DB$APP
$WEB $WEB
$WEB $WEB
$WEB $WEB
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
$SUPERVISION
$SUPERVISION
$ORCHESTRATION
$SERVICE_DISCOVERY
$ORCHESTRATION
$SERVICE_DISCOVERY
$MONITORING
$MONITORING
$PAAS
$FIXED_APP
$LEGACY $DATACENTER
$REWRITE
$SECRETS
$SECURITY
$COMPANY
$ADD-ON
$APP2
$ADD-ON
$APP3
$PAAS
$FIXED_APP
$LEGACY $DATACENTER
$REWRITE
$SECRETS
$SECURITY
$COMPLIANCE$COMPAN
Y
$ADD-ON
$APP2
$ADD-ON
$APP3
$PAAS
$FIXED_APP
$LEGACY $DATACENTER
$REWRITE
$SECRETS
$SECURITY
$COMPLIANCE$COMPAN
Y
$ADD-ON
$APP2
$ADD-ON
$APP3
$PAAS
$FIXED_APP
$ADD-ON
$APP3
$LEGACY $DATACENTER
$REWRITE
$SECRETS
$SECURITY
$COMPLIANCE$COMPAN
Y
$REWRITE
$ADD-ON
$APP2
The Looming Complexity Crisis
• Growth of connected devices due to cheap compute
• Growth of data generated by these devices
• Growth of compute required to store and process this information
The Looming Complexity Crisis
• Commit to tightly bound solutions, before you write any business logic
• Complexity must be dealt with first or it can derail you later
• Infrastructure choices dictate the design of the app
• Proliferation of app stacks and heterogenous environments
• Infrastructure nightmares
Developing “infrastructure-first”
• Business value is in the application
• We start at the bottom and work our way up
• Backwards approach with unintended side-effects (increased complexity, high error rates)
• Instead, start with the business value the work our way down
Developing Human First
• Automation that Reduce Complexity (or perceived complexity)
• Automation that Informs Humans (rather than mechanizes them)
• Automation that Reduces Error
• Automation that Liberates
98
Developing “application-first”
To be successful, the application would need to
• Know how to configure itself for its eventual infrastructure
• Have all of its behaviors and dependencies defined up front
• Know how to adapt to different topologies/environments
• Respond to dynamic changes automatically
• Act with isolation and immutability
Isolation and immutability
• For isolation, our apps are distributed as artifact
• We could expose an interface to change behavior, without changing bits
• That interface could allow dynamic tuning at runtime
• Isolate from downstream dependencies
• Infrastructure is another downstream dependency
Build on past experience
• Failure to isolate infrastructure choices from the app lead to complexity
• Runtimes (e.g. cloud, paas, containers) are rapidly evolving
• Many runtimes only focus on workload placement
• No restrictions on language or execution model
• Cross platform portability is essential
• Support for legacy apps matters
Develop in an application-first model
• Start with the behavior and business value
• Give your apps isolation and immutability
• Expose a mechanism to dynamically reconfigure
• Defer decisions about infrastructure until runtime
• Apps that automatically respond to change
• Do that anywhere
• Do that with new and legacy apps
Habitat Components
• Habitat Studio - an application packager
• Habitat Plans - instructions to download, compile, and install
Habitat Components
• Habitat Studio - an application packager
• Habitat Plans - instructions to download, compile, and install
• Habitat Depot - a place to upload and download your app packages
Habitat Components
• Habitat Studio - an application packager
• Habitat Plans - instructions to download, compile, and install
• Habitat Depot - a place to upload and download your app packages
• Habitat Supervisor - an intelligent runtime with deployment coordination and service discovery built in
Package Format (.hart)
Plan Artifact Depot
Bare Metal
Containers
AMI
VM
1. Packaging an application starts with a plan. A Plan is as simple to create as a Dockerfile
2. Start by launching the isolated Build environment called a “Studio”
a. Enter the studio (hab-studio enter)
b. Define your Plan (vim plan.sh) to define the software build from source using standard BASH
c. Define the Configuration template for the software, using Mustache (logic-less scripting language)
d. Define the Configuration defaults, using TOML (configuration language)
e. Define Hooks for supervisor behavior: initializing, running, defining a health check, etc
Habitat Workflow
Build Service
Plan Artifact DepotBuild
Service
A Build Service with a workflow to describe the software and behavior for Habitat applications
Explicit about dependenciesIncludes what is configurable about the application
Packages are developed and built in an isolated build environment (hab studio)
Built Artifacts are post-processed in one step to multiple formats including Docker Images
Source Code Repo
Habitat Technology
Depot Artifact
Supervisor: An Intelligent Run-Time Supervisor w/ REST API (hab-sup)
Supervisor Supervisor
Supervisor Supervisor
Ring
Leader
Initializer
Stand Alone
Part of an Artifact, the Supervisor manages Application node behavior at runtime, providing automation and intelligence:Utilizes predefined Plan to manage application behaviorBuilt in service discoverySelf-organizes with other Supervisor nodes on a Ring into topologiesRings have no single point of failure
Supervisor executes a variety of hooks for defined behavior:
Init, run, health_check, reconfigure, file_update
Supervisor is accessible via REST API:
Habitat Technology
Security & Management
Secret Payloads
Service Service
Service Service
Pub Key
Symmetric Encryption
Service Service
Service Service
Load Balancer
Enterprise-grade security allows for secure runtime management of configurations and payloadsEncrypted, authenticated run-time configurationAutomatic, safe, atomic software updates based upon policy (example: Blue/green deployments)Dynamic topology updatesHandles inter-service discovery through binding
113
Habitat Technology
Habitat Technology
Plan Artifact DepotBuild
Service
Supervisor Supervisor
Supervisor Supervisor
Ring
Bare Metal
Containers
AMI
VM
Depot
Build Service & Workflow
Package Format Artifact Distribution Server
Artifact Distribution Server
Intelligent Run-Time Supervisor w/REST API
Source Code Repo
Applications run as distributed, fully automated, and capable autonomous actors
Immutable applications, but flexible and easy to manage because automation travels with the application
Bundles what your apps need to run and nothing else
Embedded automation choreographs application cluster topology/behavior
A network with no reliance on external services and no single-point-of-failure
Provides continuous deployment without traditional Application Release Automation (ARA) tooling
The solution is the same:
Habitat’s approach
HabitatApplication automation that enables modern
application teams to build, deploy, and manage any application in any environment - from
traditional data-centers to containerized microservices.
> Operating System centric; application automation becomes complex
> Focused on integrated SDKs; “Cloud Native” rewrites
> Building tools around applications; pushing in the old direction
> Focuses on the grid as the unit of automation; apps must be specialized
> Focuses on the container as the unit of automation; apps are opaque.
Wait, doesn’t X do that?
We believe that all applications can be autonomous across any platform
PlanCollaborate
Inspect Build
Try Habitat for yourself
• https://www.habitat.sh/try
• Tutorials
• Getting started guide
• Extensive documentation
• Open Source!
– https://github.com/habitat-sh/habitatThank you & t-shirts!