nyc amazon web services meetup: how glue uses aws

AdaptiveBlue @AWS NYC Meetup

June 4, 2009

Mark Tabry, Software Engineerhttp://getglue.com

Agenda

About AdaptiveBlue Glue: The Network of People and Things Glue: Building on Amazon Web Services Glue: Semantic Technology Stack

About AdaptiveBlue

Founded in 2006, based in New York

Focuses on enhancing browsing experience

Launched BlueOrganizer and Glue add-ons forFirefox and SmartLinks Widgets for blogs

Get Glue. The Network ThatSticks With You.

http://getglue.com

What is Glue?

Glue is a contextual network that usessemantic technology to automaticallyconnect people around everyday things -books, music, movies, stars, artists,stocks, wine, restaurants and more.

1. Contextual: Glue is distributed and appearswhen it makes sense on popular sites.

2. Automatic: Users participate in Glue just bybrowsing their favorite sites.

3. Simple: Glue removes the friction involvedin networking - the network comes to you.

Glue Demo

Glue:Building on Amazon

Web Services

AWS-based Architecture

Host N (EC2)

Amazon SimpleDB

Interactions betweenPeople and Things

Rackspace MySQL

User accountsAnalytics

Glue Web Service

Host 1 (EC2)

Glue Web Service

Batch Services

. . .

Amazon S3

Object Database/People Profiles

Database Layer

Web Service Layer

Load Balancer 1 Load Balancer 2

Round Robin DNSLoad Balancer Layer

Browser Add-Ons Widgets iPhones Facebook Apps API Clients

Client Layer

Batch Services

AdaptiveBlue AWS Stack

Relating People and Things ( SimpleDB )

Records of people’s interactions around things are stored in SimpleDB Domains using duplication for fast access.

Transactional and Batch Support ( EC2 )

Web Service Requests and batches are distributed through EC2 instances.

Storing Object Meta Data ( S3 )

XML representation of millions of books, music, movies, etc. is stored using Amazon S3

Client

Idea:

Create flat database with auto-indexed tables.

Main Features:

Each attribute is indexed. Record structure is flexible. Basic operators in queries Supports sorting.

Simple DB DomainRecord 1

Put recordGet recordQuery records

Key1 Attributes: A1,A2…

Record NKey2 Attributes: A1,A2…

…

Amazon SimpleDB in a Nutshell

Object Domains

Interaction RecordKey1 Attributes: A1,A2…

How Glue uses SimpleDB

Each record is duplicated into Object and Person Domain The Key is a combination of USER_ID and OBJECT_KEY Djb2hash is used to calculate the domain for each record

Records for each USER and each OBJECT inside the same domain.

OD2OD1 ODN…

People Domains

PD2PD1 PDN…

Client

Idea:

Put/Get objects into bucketsbased on unique keys.

Main Features:

Public/Private access. Support for large objects.

Amazon S3

Bucket 1 Bucket N…

Put object Get object

Amazon S3 in a Nutshell

Object Bucket

How Glue Uses S3

XML-files with object information

People Bucket

XML-files with user and friends info

XML is serialized as string and written to S3 Each file has a unique key: OBJECT_ID or USER_ID/profile, etc.

MachineImage

(OS + Apps)

Usage:

Create Machine Image Deploy the image to S3 Start 1 or more instances Use it as regular machine(s)

Main Options:

Dynamic/Static IPS Choose cores Choose locations Persistence via EBS

Amazon EC2 in the Nutshell

How Glue uses EC2

Host N (EC2/Rackspace)

Glue Web Service

Host 1 (EC2/Rackspace)

Glue Web Service

Batch Services

. . .

Load Balancer 1 Load Balancer 2

Round Robin DNS

Batch Services

Web Service processes transactional requests Batch Services are time-based & run on sets of USERS and OBJECTS

The system scales by equally partitioning Data and Requests

http://getglue.com

http://twitter/marktabryMark.Tabry@gmail.com