hackfmi talk #3 - interesting times: 6 current paradigm shifts in it - teodor tonchev
DESCRIPTION
Teodor Tonchev, from VMWare Bulgaria, with education from MIT and 18 years of experience in the IT industry, talked about the radical changes in the industry, that will change the way we think about and create Software. From Virtualization, Cloud, App Development to Polyglot Persistence Layers and Software Defined DatacentersTRANSCRIPT
© 2013 VMware Inc. All rights reserved
Interesting Times: Paradigm Shifts in IT
Teodor Tonchev, Staff Engineer, VMware
2
Definition
Paradigm Shift:
A change from one way of thinking to another
3
Virtualization
4
Create Virtual Machines
Run different Operating
Systems and Applications
within these VMs
Clone/Move/Backup VMs
Snapshot/Restore VMs
VMware Workstation™
Virtual Machines
Chang
ed
script
and
slide
OS
APP
OS
APP
OS
APP
OS
5
Hypervisor – an Operating
System for running VMs
Many servers now come
with ROM-based hypervisor
VMware ESX™
Server Virtualization
Chang
ed
script
and
slide
OS
APP
OS
APP
OS
APP
6
OS
APP
vMotion allows running VMs
to be moved among servers
seamlessly
Clients connected to the
VMs do not observe any
changes VMware vSphere™
VMware vMotion
Chang
ed
script
and
slide
7
OS
APP
OS
APP
OS
APP
DPM consolidates
workloads onto fewer
servers when the cluster
needs fewer resources
Places unneeded
servers in standby
mode
Brings servers back
online as workload
needs increase
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
VMware vSphere™
VMware DPM (Distributed Power Mgmt)
DPM powers off
server when
requirements
are lower
DPM brings
servers back
online when
load increases
Chang
ed
script
and
slide
8
Distributed Resource Management (DRS)
Shrink and grow of
applications based on
demand and priority
Dynamic and responsive
load balancing VMware vSphere™
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
9
Current Virtualization Use
2012 Survey: 51% of x86 Servers Now Virtualized
Nearly 92 percent of all enterprises use at least some form of
virtualization.
Majority of servers on the market support integrated hypervisor
10
Cloud
11
What is Cloud Computing?
Where at home do you keep your Electric Power Station?
Cloud: Computing resources provided on demand, as a service,
remotely (Internet or Intranet)
• Utility-Based Computing is another term for cloud computing
12
Cloud Categories
13
Starting a software service is much easier now
Application Server
OS
App Libs
DB
With IaaS there is no need to buy hardware.
With PaaS one can focus only on developing the App.
14
Application Development
15
Pets vs. Cattle
16
Multicore Future
Chips are reaching the physical limits
Moore’s law still holds, but due to increasing the number of cores
Now: tens, hundreds of cores
Soon: 1000s of cores
2020s: Millions of cores
17
Parallel programming
Approach 1: Shared Memory
• Threads
• Locks/Mutexes
DOES NOT SCALE. COMPLEX (e.g. error handling).
Approach 2: Message Passing
• Actors
• Messages
SCALES WELL. RELATIVELY SIMPLE (once you learn it).
18
Actor Model
19
Actor Model
20
Actor Model
Languages
• Erlang
• Scala
Frameworks
• Akka (Java/Scala)
• Kilim (Java)
• Pulsar (Clojure, Python)
• …
Erlang Hall of Fame:
• Availability: 99.9999999%
21
Databases
22
Background
Limited data
Localized data
Homogeneous requirements
RDBMS have matched these requirements
23
The Rise of NoSQL
ACID
• Atomic
• Consistent
• Isolated
• Durable
vs
BASE
• Basic Availability
• Soft-state
• Eventual Consistency
24
Requirements Have Changed
Mass scale (Big Data)
Distributed Data Storage
Different varieties of data
Incompatible requirements
RDBMS are no longer suitable to handle all use cases
25
Many Databases
26
NewSQL strikes back
Eventual Consistency complicates life a lot
"We also have a lot of experience with eventual consistency systems at
Google. In all such systems, we find developers spend a significant
fraction of their time building extremely complex and error-prone
mechanisms to cope with eventual consistency and handle data that
may be out of date.“
MIT Prof. Michael Stonebraker: “The Traditional RDBMS Wisdom is
All Wrong”
• Most OLTP systems can afford to buy the amount of memory needed to keep
data off the disk.
• Single-threading gets rid of the latching overhead.
• OLAP: Column Stores are 50-100 times faster
• …
27
Memory/Network/Disk speed
28
Separate CPU/Memory/Storage Racks
140 TB
29
So Who is Right? NoSQL or NewSQL?
Well… Both!
Polyglot Persistence
30
“Software Defined Datacenter”
20.11.2013 г. 30
31
Storage
32
Interesting Times
Application/OS Deployment
• Deploy directly on the hardware
• New business -- buy hardware
Application Development
• Pet Apps (Fixed Scale)
• Sequential Programming + Parallel
Programming with Shared Memory
Databases
• Local fast disks
• RDBMS or NoSQL
Data centers
• Specialized hardware
• Virtualize the hardware
• Use a service
• Cattle Apps (Dynamic Scale)
• Parallel Programming with
Message Passing
• In-memory, distributed
• Polyglot Persistence
(NoSQL,NewSQL,etc)
• Cheap hardware + software