unlocked london - technical track

No 11 cavendish

square

11 july 2013

london

Thursday, July 11, 13

developer VOCAB

Alex brandtsr developer


MODULES

3


LIBRARIES

4


FRAMEWORKS

•The building blocks of an application - a set of common software functions that take the work out writing all code from scratch.

5


6


7


8


LANGUAGES

9


10


APIs

11


APIs

12


What is a data structure?

•images• url

• stars

• *title

• caption

• variants

•small

•medium

•big

•tags

•*_id

•rolestars•*image_id

•*role_id

•roles• *name

• privileges

• login

• description

• location

• child_roles

• _id

13


API tier/Application

STORAGE

APPLICATION

PRESENTATION

14


API tier/Application

STORAGE

APPLICATION

PRESENTATION

APPLICATION/API

14


Bindings

15


Bindings

16


Bindings

17


EVERYTHING IS CODE

wayne wallscloud evangelist

alex brandtsr developer


Infrastructure as code

19


20


21


Chef Server

Chef Workstation

Chef Client

knife

node

Chef Client

node

environment

22


•Nodes: Servers

•Environments: Groups of nodes (Dev/QA/Prod)

•Recipes: Describes the desired state of a node

•Cookbooks: Group of Recipes

•Roles: Group of Cookbooks (Web)

•Attribute: information about the current state of a node

23


24


PuppetCA

PuppetMaster

PuppetAgent

https

PuppetAgent

PuppetAgent

25


•Nodes: Servers


•Manifests: Describes the desired state of a node

•Catalog: Group of Manifests

•Facts: information about the current state of a node

26


27


SaltMaster

SaltMinion

SaltMinion

SaltMinion

28


•Nodes: Minions


•States: Describes the desired state of a node

•Grains: information about the current state of a node

29


Thinking like a developer30


scrum Development

Product Backlog

Sprint Backlog Sprint Working increment

of the software

31

1. Design

5. QA 2. Build

4. Release 3. Configure


scrum Development

32

1. Design

5. QA 2. Build

4. Release 3. Configure


TYPES OF DEVELOPMENT

33



33

Test Driven Development



33


Behavior Driven Development



33


Behavior Driven Development Domain Driven Development



33



Application Bliss

Domain Driven Development


Model View Controller (MVC)

UPDATES MANIPULATES

SEES USES

USER

MODEL

VIEW CONTROLLER

34


DEv tools


36


37


Gerrit

38


39


CODE REVIEW PROCESS


Our hero, the developer, submits code for review.

The code is reviewed in Gerrit by other developers, who approve the change.

Jenkins runs automated tests against the code change.

Jenkins pushes the code to GitHub so others can build on the change.

41


POLYGLOT PERSISTENCE


technical track



A database, in the most general sense, is an organized collection of data.

Database Defined

http://glenn-packer.net

43


http://glenn-packer.net/

http://glenn-packer.net/

ACID

•Atomicity: refers to a database’s ability to either fully process or fully roll back a transaction

•Consistency: database should ensure that all data written therein follows all rules and constraints specified in the database

•Isolation: transactions must be processed securely and independently, without interfering with each other

•Durability: the database must ensure that all committed transactions are saved permanently and cannot be accidentally erased, even in a database crash

44


•A relational database is essentially a group of tables or, to use the technical name, entities.

•Each table is made up of rows (tuples) and columns (attributes).

relationaldatabases

45


Some companies needed something more to accommodate their growing data

rdbms may not be the best fit

46


Enter Non-Relational database

management systems

47


INTRODUCTION TO NOSQL

• NoSQL Database (1998)• NoREL• “Not only SQL”• NoSQL (2009)

48


NoSQL is difficult to define but most NoSQL databases will share common characteristics

1. Non-relational

2. Distributed

3. Open-source

4. Horizontally Scalable

5. Schema-less

NoSQL

49


An “aggregate” is a collection of related objects that we wish to treat as a unit

orders

customers

order lines

credit cards

ID: 1001

line items:

customer: Ann

0321293533

0321601912

0131495054

2

1

1

$48 $48

$39

$51

$39

$51

payment details:

Card: AmexCC #: 12345expires: 04/2001

aggregate

50


There are several different types of data stores that are

considered to be NoSQL

•Key/Value Databases

•Document Databases

•Column-Family Stores

•Graph Databases

51


Key/Value Database•Primarily constructed of aggregates that have a Key or ID that’s used to access the data

•The database cannot see structure within the aggregate

52


Key/Value Database

•Good Use Cases:

• Storing Session Information

• User Profiles & Preferences

• Shopping Cart Data

• Bad Use Cases:

• Data Relationships

•Multi-operation Transactions

•Query by Data

53


•Like a Key-Value database, document databases primarily store and access data using aggregates

•In contrast to the Key-Value database, document data bases can see structure within the aggregate.

•You are able to retrieve the whole aggregate or specific parts of it

document Database

54


document Databases

•Good Use Cases:

• Event Logging

• Blogging Platforms

• Real-time Analytics

• Bad Use Cases:

• Complex Transactions

•Queries against Varying Aggregate Structures

55


• Can be looked at as a two-level aggregate structure

• The first key is often described as a row ID picking up the aggregate of interest

• The second level values of the aggregates are referred to as columns

• You are able to access a row as a whole or operations will allow you to pick out a particular column to get specific data

COLUMN FAMILY

56


COLUMN-FAMILY datastores

•Good Use Cases:

• Event Logging

• Blogging Platforms

• Counters (page visits)

• Expiring Usage

• Bad Use Cases:

• ACID transactions required

• Early prototype applications

57


•Unique compared to most NOSQL types because it was not developed with the intention of scaling horizontally into a cluster

•Data is stored as “nodes” that are connected by edges

•Edges are directional in nature and must be specified

•Queries need a starting point and them will traverse the edges to gather the data requested

graph database

58


Amanda

KNOWS

KNOWS

KNOWS

Robert

Gerry

Lamar

KNOWS

App 1

BUILTBUILT

KNOWS

App 2

USES

BUILT

Christie

KNOWS

Destiny

Jessica

KNOWS

App 3

BUILT

USES

USES

USES

KNOWS

graph database

59


graph databases

•Good Use Cases:

• Connected Data

• Location-based services

•Recommendation engines

• Bad Use Cases:

• Ability to update specific aggregates

60


Companies using nosql

61

DATABASE COMPANIES


THE FUTURE IS POLYGLOT

62

Neo4Jdatabase

Cassandradatabase

Couchdatabase

MongoDBdatabase

RELATIONALDATABASE

Graph-structureddomain rules

Columnar dataaccess with

decentralization

Documentstructures

Document structureswith offlineprocessing

(Actors) (Actors)

(Actors)(Actors)

Asynchronous message passing

MODULE 1

MODULE 2

MODULE 3

MODULE 4

IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg


63

POLYGLOT PATTERN: cache

ORDERTAKING

RESTAURANTMANAGEMENT

REDISCACHE

MYSQL DATABASE SECONDFIRST

CONSUMERRESTAURANT

OWNER


64

POLYGLOT PATTERN: write to sql and NOSQL

NEWTWEET

INSERT

LPUSHXLTRIM

MySQL

TWEETS

FOLLOWS

FOLLOWERS

Redis: Timeline

FOLLOWER 1

FOLLOWER 2

FOLLOWER 3

Avoidexpensive

MySQL joins


65

POLYGLOT PATTERN: REPLICATE FROM MYsql to NOSQL

QUERYSERVICE

UPDATESERVICE

REDISMYSQL

DATABASE

SYSTEMOF RECORD

MATERIALIZED VIEW

READER WRITER

replicate and

denormalize

query() update()


66

Atomic

Consistent

Isolated

Durable

ACID to base

Basically Available

Soft state

Eventually consistent


5 PILLARS OF CLOUDINESS



technical track


FIVE PILLARS OF CLOUDINESS

1 2 3 4 5

Parallel Modular Horizontal Agile Secure

68


Parallel


Parallel - what

1MRecords

Answer

Node

70


Parallel - how

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M

71


Parallel - how

72

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M


Parallel - how

73


Parallel - how

DATA

CALCULATIONMAP

CALCULATIONREDUCE

DATA

CALC

D

C

R

D

D

C

R

D

D

C

R

D

SHUFFLE

DATA

73


Avoid Deadlock

74


Thread 1

Deadlock

75

Thread 2


File A

Thread 1

Deadlock

75

Thread 2


File A File B

Thread 1

Deadlock

75

Thread 2


File A File B

Thread 1

Deadlock

75

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

75

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

75

X

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

75

X X

File B

Thread 2


Solution? Scheduling

Y

XA

B

A

B

Y

X

1

2

3

4

76


Parallel - why

1M records

Answer

Node

1M records

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node750K-1M

77


Parallel - why

78


Parallel - why

79


modular


ONE PERSON BUILDS A CAR

Modular - whatExample: Assembly Line

81


CHASSIS ENGINE BODY INTERIOR

DecoupledSelf-contained unitsData Communication

Modular - whatExample: Assembly Line

82


Modular - what

Modules within Modules


83

Nested Modularity



Modular - what

84

Modules within ModulesNested Modularity



Modular - what

SEATS FLOOR

DASHBOARD

GASAND

BRAKE PEDALS

84

Modules within ModulesNested Modularity


Small units are more maintainableAllows us to tolerate some partitions (but not all)

Allows us to be parallel!


modular - Why

85


Direct communication (1:1)Topic communication (1:N)

Fanout communication (1:ALL)

modular - Why

86


•Decouple requests

•Message Queue

•Make small units of work

•Define tiers by function

modular - how

87


Potential for Upstream Complexity


88


defect



88


defect defect



88


defect defect defect



88


defect defect defectdefect



88


defect defect defectdefect FAIL



88


defect defect defectdefect FAIL

Mandelbugs



88


horizontal


2 GB2 CORES

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

VERTICAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

SHARED NOTHING THROUGH STATELESSNESS

Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL


Horizontal - What

90

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL


Horizontal - What

2 GB2 CORES

90

8 GB4 CORES


Horizontal - why

91


Horizontal - why

92


ONE SERVER NEAR

CAPACITY

Horizontal - why

92


ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

Horizontal - why

92


ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

Horizontal - why

93


1234 MESSAGE QUEUE

94

Horizontal - HOW


1234MESSAGE QUEUE

95

Horizontal - HOW


1

2

3

4

96

MESSAGE QUEUE

Horizontal - HOW


1

2

3

4

96

56789101112 MESSAGE QUEUE

Horizontal - HOW


97

5678 9101112MESSAGE QUEUE

1

2

3

4

Horizontal - HOW


97

5678 9101112

ADD MORE SERVERSBASED ON

MESSAGE QUEUE

MESSAGE QUEUE

1

2

3

4

Horizontal - HOW


MESSAGE QUEUE

98

5

6

7

8

9

10

11

12

13141516


MESSAGE QUEUE

1

2

3

4

Horizontal - HOW


99

5678 9101112

13141516


MESSAGE QUEUE

1234

MESSAGE QUEUE

Horizontal - HOW


100

5678 9101112

13141516


MESSAGE QUEUE

1234

MESSAGE QUEUE

Horizontal - HOW


101

5678 9101112

13141516


MESSAGE QUEUE1234

MESSAGE QUEUE

Horizontal - HOW


101

5678 9101112

13141516


MESSAGE QUEUE1234

MESSAGE QUEUE

Horizontal - HOW

DECREASE SERVERSBASED ON

MESSAGE QUEUE


MESSAGE QUEUE

102

5678 9101112

13

14

15

16


MESSAGE QUEUE1234

Horizontal - HOW

DECREASE SERVERSBASED ON

MESSAGE QUEUE


agile


Agile - WHAT

the power of moving quickly and easily; nimbleness

104


Features In Progress Blocked Tabled Testing Complete

Feature A

Feature B

Feature C

105

Agile - why Monday


Agile - why


Feature A

Feature B

Feature C

106

Tuesday


Agile - why


Feature A

Feature B

Feature C

Feature E

Feature D

106

Tuesday



Feature AFeature BFeature C

Feature E

Feature G

Agile - why

107

Wednesday



Feature AFeature BFeature C

Feature E

Feature D

Feature F

Feature G

Agile - why

107

Wednesday



Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Agile - why

108

Friday



Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Feature H

Feature I

Agile - why

108

Friday



Feature A

Feature B

Feature C

Feature EFeature D

Feature F

Feature G

Feature H

Feature I

Agile - why

109

Friday - WK 2


Agile - why

57%

14%

29%

Failed Successful Challenged

49%

42%

9%

Waterfall Agile

Source: The CHAOS Manifesto, The Standish Group, 2012

110


• CI/CD when appropriate

• Quality is in the eye of the beholder

• Release fast, release slow

Agile - How


111


IMAGE: blogs.bu.edu

Agile - How

• Increase quality by reusing code from partners and libraries

112


Agile - How

113

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


ENSURES QUALITY

Agile - How

113

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


ENSURES CONFIDENCE

Agile - How

114

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


secure


Security - what

116

•All about Trust•Provider•Customer•User


117

SECURITY - HOW•Security at every layer

PRESENTATION

APPLICATION

STORAGE


117

SECURITY - HOW•Security at every layer

PRESENTATION

APPLICATION

STORAGE

INTERFACESECURITYACROSS ALL LAYERS


118

SECURITY - HOW

•Use industry standards

•SSL

•Encryption

•At rest

•In-transit


119

SECURITY - HOW•Authentication

•User authentication

•Manage yourself

•Offload to 3rd party

•Host-based Authentication

•Security Groups

•Firewalls

•Configuration Management Systems


120

SECURITY - HOW•Hybrid Cloud

•Store different types of data in the right place

•Sensitive data in on-premise cloud

•Non-sensitive on public cloud


121

SECURITY - WHY

•Trustworthiness as an organization

•Cloud security is too important to leave up to the end-user alone

•Ownership gaps that leads to process and governance issues

•Compliance

•Logging and auditing

•Regulatory requirements


BREAK

3:00 - 3:15


APPLICATION dissection


technical track



Data Schema

•images• url

• stars

• *title

• caption

• variants

•small

•medium

•big

•tags

•*_id

•rolestars•*image_id

•*role_id

•roles• *name

• privileges

• login

• description

• location

• child_roles

• _id

124


Requirements•List stars

•List roles’ images

•CRUD

•Tag searching

•Who liked what

•Social media

•Image resizing

•Added by tag

•Hot tagged images

125


application decisions

API TIER

UT UR RSWORKERTIER

LB

AUTH

MQ DB

OBJECTSTORAGE

CDN

SOCIALMEDIA

126

INTERNET

•Load Balancer•Infrastructure•Authentication•Message Queue•Database•Content Delivery•Social Media



API TIER

UT UR RSWORKERTIER

AUTH

MQ DB

OBJECTSTORAGE

CDN

SOCIALMEDIA

126

INTERNET




API TIER

UT UR RSWORKERTIER

MQ DB

OBJECTSTORAGE

CDN

SOCIALMEDIA

126

INTERNET




API TIER

UT UR RSWORKERTIER

DB

OBJECTSTORAGE

CDN

SOCIALMEDIA

126

INTERNET




API TIER

UT UR RSWORKERTIER

OBJECTSTORAGE

CDN

SOCIALMEDIA

126

INTERNET




API TIER

UT UR RSWORKERTIER

CDN

SOCIALMEDIA

126

INTERNET




API TIER

UT UR RSWORKERTIER

CDN

126

INTERNET




API TIER

UT UR RSWORKERTIER

126

INTERNET



PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

127


•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

127


•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

•Works in tandem

127


•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

WORKERTIER

API TIER

•Works in tandem•Doesn’t wait for workers

127


•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

•Update Tags, Update Relations, Resize Workers operate in tandem

WORKERTIER UT UR RS


127


•API Tier

PARALLEL

API TIER

UT UR RS

INTERNET

•Update Tags, Update Relations, Resize Workers operate in tandem

•Non-blocking actions

WORKERTIER UT UR RS

API TIER


127


MODULAR

API TIER

UT UR RSWORKERTIER

INTERNET

128


•Separate tiers by functionality

MODULAR

API TIER

UT UR RSWORKERTIER

API TIER

UT UR RS

INTERNET

128



MODULAR

API TIER

UT UR RS

•Use MQ to decouple API from worker

WORKERTIER

INTERNET

128



MODULAR

API TIER

UT UR RS

•Use MQ to decouple API from worker

•Separate workers to simplify units of work

WORKERTIER UT UR RS

INTERNET

128


Horizontal

API TIER

UT UR RSWORKERTIER

INTERNET

129


•Restful API for shared nothing scale

Horizontal

API TIER

UT UR RSWORKERTIER

API TIER

INTERNET

129



Horizontal

API TIER

UT UR RS

•Workers share no state, simply work on input

WORKERTIER

INTERNET

129



Horizontal

API TIER

UT UR RS

•Workers share no state, simply work on input

•Scale data across nodes (even DCs)

WORKERTIER UT UR RS

API TIER

INTERNET

129


AGILE

API TIER

UT UR RSWORKERTIER

INTERNET

130


•Don’t re-invent ability or hosting

AGILE

API TIER

UT UR RSWORKERTIER

INTERNET

130



AGILE

API TIER

UT UR RS

•Split processes that are individually testable

WORKERTIER

API TIER

UT UR RS

INTERNET

130



AGILE

API TIER

UT UR RS


•CI/CD is achievable by component

WORKERTIER

INTERNET

130



AGILE

API TIER

UT UR RS


•CI/CD is achievable by component

WORKERTIER

API TIER

INTERNET

130


SECURE

API TIER

UT UR RSWORKERTIER

INTERNET

131


•Use HTTPs for secure data transmission

SECURE

API TIER

UT UR RSWORKERTIER

API TIER

INTERNET

131



SECURE

API TIER

UT UR RS

•No host logins; managed by Chef

WORKERTIER

API TIER

UT UR RS

INTERNET

131



SECURE

API TIER

UT UR RS

•No host logins; managed by Chef

•StormPath for user authentication

WORKERTIER

INTERNET

131


thank you!@rackspaceUK@Racker_beighton@waynewalls@alunduil@jennbolestweets


unlocked london - technical track

Technology

types of development

nosql nosql database

code change

node cookbooks

database isolation

database crash

desired state

node catalog