Slide 1

Storing and Accessing Live Mashup Content in the CloudKrzysztof Ostrowski, Ken Birman

Cornell University{krzys|ken}@cs.cornell.edu1Agenda2A new, versatile storage abstraction:Checkpointed Channel (CC)

A new web application architecture:A web of (hyperlinked) CCsAgenda3Introduction4

Storing Content in the Cloud

server in adata centerclient-serverinteractionscloudcontentedge5

Storing Content in the Cloud

server in adata centerclient-serverinteractionscloudcontentedge6

server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge7

server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge8

server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge9

server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge10

server in adata centerclient-serverinteractionscloudStoring Content in the Cloudcontentedge11

Storing Content in the Cloudcontentclient-serverinteractionsserver in adata centerupdate relayed through the server12

server in adata centerStoring Content in the Cloudexplicit request,e.g. HTTP GETcontentcloudedge13http://liveobjects.cs.cornell.edu

14

server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicapeer-to-peerinteractions15

server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica16

server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica17

server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replica18

server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replicaupdated19

server in adata centerStoring Content at the Edgeedgepeer-to-peerinteractionscontent replicacontent replicacontent replicaupdatedmulticast20

server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicaupdatedupdatedupdated21

server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicareplicated statemachine22

server in adata centerStoring Content at the Edgeedgecontent replicacontent replicacontent replicamanagemembership23

server in adata centerStoring Content at the Edgeedgedatacontrol24

server in adata centerStoring Content at the Edgeedgedatacontrol25Cloud vs. Edge?

server in adata centeredgepeer-to-peerinteractionscontent replicacontent replicacontent replica

server in adata centerclient-serverinteractionscloudcontentedge26

server in adata centerCloud vs. Edge?edge27

server in adata centerCloud vs. Edge?edge28

server in adata centerCloud vs. Edge?edge29

server in adata centerCloud vs. Edge?edgeno replicas left30

Cloud vs. Edge?

server in adata centerclient-serverinteractionscloudcontentedgeflow ofupdatesbottleneck31

Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge

flow ofupdatesbottleneck32

Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge

flow ofupdatesbottleneck33Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedge

flow ofupdates

bottleneck34Cloud vs. Edge?server in adata centerclient-serverinteractionscloudcontentedgeflow ofupdates

bottleneck

35Cloud vs. Edge?Server capacity:SecondLife: ~40 clients/serverSecond Life and the New Generation of Virtual Worlds.S. Kumar et al. 2008. IEEE Computer, 41(9):46-53

36Cloud vs. Edge?Server capacity:SecondLife: ~40 clients/server

virtualisland37Cloud Edge?

server in adata centeredgepeer-to-peerinteractionscontent replicacontent replicacontent replica

server in adata centerclient-serverinteractionscloudcontentedgepersistencescalability+and38

Extending Cloud to the Edgeserveredgecloud39

Extending Cloud to the Edgecloudserveredge40

peer-to-peerinteractionsclient-serverinteractionsExtending Cloud to the Edgecloudservercontentcontent41

peer-to-peerinteractionsclient-serverinteractionsExtending Cloud to the Edgecloudservercontentdistributedprotocol(client-server,peer-to-peer, etc.)content42

Extending Cloud to the Edgecloudserveredge43There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft

=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud44There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft

=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud45There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft

=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud46There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft

=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud47There are:40+ times more PCs than servers(billions today, to double in five years)2000+ client PCs per Google server40,000+ times more client PCs purchased each year by home users than new servers purchased each year by Microsoft

=> a lot of computational power and resources at the edge that are greatly underutilizedEdge is Larger than the Cloud48Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction49Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction50Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction51Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction52Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction53Unify different content access models:Centralized vs. replicatedCentralized: web service, databaseReplicated: distributed P2P replication protocolsPersistent vs. temporaryPersistent: server in a data centerTemporary: collaboration session formed ad hocServer-side vs. client-sideShared public vs. shared privatePublic: web service or collaboration sessionPrivate: session state, cookies, etc. Stateful vs. statelessStateful: variables, files, objects, database tablesStateless: video or notification streams

New Storage Abstraction54http://liveobjects.cs.cornell.edu

55Checkpointed Channels (CC)56

Architecture57

Architecture58

Architecture59Architecture

60Architecture

61Architecture

networkmessages62the edge may cut through the channel

content physically resides outside the channelbut it could also be cached in it63the edge may cut through the channel

content physically resides outside the channelbut it could also be cached in it64

content physically resides outside the channelbut it could also be cached in itthe edge may cut through the channel65

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates66

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates67

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates68

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates69

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates70

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates71

Dynamics (local)Checkpointed Communication Channel (CC)UCUUUinitial checkpoint72

Dynamics (local)Checkpointed Communication Channel (CC)Cinitial checkpointmultiple updatesUUUU73

Dynamics (local)Checkpointed Communication Channel (CC)Cinitial checkpointmultiple updatesUUUU74

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates75

Dynamics (local)Checkpointed Communication Channel (CC)UupdatesUU76

Dynamics (local)Checkpointed Communication Channel (CC)UupdatesUU77

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates78

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates

79

InterfacesCheckpointed Communication Channel (CC)a writable stream of checkpoints and updates80

Dynamics (local)Checkpointed Communication Channel (CC)requestcheckpointR81

Dynamics (local)Checkpointed Communication Channel (CC)requestcheckpointR82

Dynamics (local)Checkpointed Communication Channel (CC)Ccheckpoint83

Dynamics (local)Checkpointed Communication Channel (CC)Ccheckpoint84

Dynamics (global)Checkpointed Channel (CC)85

Dynamics (global)Checkpointed Channel (CC)connect86

Dynamics (global)Checkpointed Channel (CC)connectinitialize87

Dynamics (global)connectinitializeneedcheckpoint88

Dynamics (global)initializeneedcheckpointR89

Dynamics (global)request checkpointneedcheckpoint90

Dynamics (global)request checkpointR91

Dynamics (global)initializerequest checkpointneedcheckpointR92

Dynamics (global)request checkpointneedcheckpointprovide checkpoint93

Dynamics (global)request checkpointprovide checkpointCcheckpoint94

Dynamics (global)provide checkpointCcheckpoint95

Dynamics (global)Ctransfercheckpoint96

Dynamics (global)transfercheckpointC97

Dynamics (global)Ccheckpointtransfercheckpoint98

Dynamics (global)Ccheckpointdeliver checkpoint99

Dynamics (global)deliver checkpointCcheckpoint100

Dynamics (global)consumecheckpointCdeliver checkpoint101

Dynamics (global)consumecheckpointCwants toupdate102

Dynamics (global)consumecheckpointCwants toupdateupdateU103

Dynamics (global)wants toupdateupdateUrequestupdate104

Dynamics (global)updaterequestupdateU105

Dynamics (global)Upropagateupdate106

Dynamics (global)propagateupdateUUU107

Dynamics (global)propagateupdateUUUupdateU108

Dynamics (global)UUUupdateUdeliverupdate109

Dynamics (global)UUUUdeliverupdate110

Dynamics (global)deliverupdateUUUU111

Dynamics (global)UUUUconsumeupdate112SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate113SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate114SemanticsUCC'+=earlier checkpointor statenewcheckpointor stateincrementalupdate115

SemanticsCkcheckpointmultiple updatesU1U2Un116

SemanticsCkU1U2Un117

SemanticsCkU1U2UnCk+1+=CkU1118

SemanticsCkU1U2UnCk+1Ck+2+=Ck+1U2119

SemanticsCkU1U2UnCk+1Ck+2Ck+n+=Ck+n-1Un120

SemanticsCkU1U2UnCk+1Ck+2Ck+n121

SemanticsCkU1U2UnCk+1Ck+2Ck+n122

SemanticsC1U2U3U5C4C6123

SemanticsC1U2U3U5C4C6C2C3C5124

SemanticsC1U2U3U5C4C6C2C3C5125

SemanticsC1U2U3U5C4C6C2C3C5126

SemanticsC1CnC2C3mastersequence127

SemanticsC1CnC2C3mastersequenceC1Un-1U1U2submitted updates128

SemanticsC1CnC2C3mastersequenceC1C5C8C9C15every application observes a subset of the master sequence129

SemanticsC1CnC2C3mastersequenceC1C5C8C9C15every application observes a subset of the master sequenceand nobody lags behind130TypesCheckpointed Communication Channel (CC)can be classified based on:the type of checkpoints,the type of updates,(and many other factors we wont dicuss)131TypesCheckpointed Communication Channel (CC)can be classified based on:the type of checkpoints,the type of updates,(and many other factors we wont dicuss)132

TypesCUXML documentsstandardized edits on XML documentsXML Channel (XC)133

TypesCUXML documentsstandardized edits on XML documentsXML Channel (XC)134

TypesCUstandardized edits on XML documentsXML Channel (XC)XML documents135

TypesCUXML Channel (XC)XML documentsstandardized edits on XML documents136TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection137TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection138TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection139TypesCheckpointed Channel (CC)Text ChannelBinary ChannelXML Channel (XC)RSS ChannelXHTML ChannelXAML ChannelReference ChannelCollection140http://liveobjects.cs.cornell.edu

141Live Distributed Objects (LO)142Ordinary Web ApplicationsdisplayUI components

143Ordinary Web ApplicationsdisplayUI components

144Ordinary Web ApplicationsdisplayUI components

145

Ordinary Web ApplicationsdisplayUI components146

Ordinary Web ApplicationsdisplayUI components147Ordinary Web ApplicationsdisplayUI components

148Ordinary Web Applications

side by sidenested149Ordinary Web Applications

150Live Objects Applicationsside by sidenested

front-end to back-end151Live Objects Applicationsside by sidenestedfront-end to back-end

datapipeline152Live Objects Applicationsside by sidenestedfront-end to back-end

datapipelineUI component153Live Objects Applicationsside by sidenestedfront-end to back-end

datapipelineUI componentMPEG-2 decoder154Live Objects Applicationsside by sidenestedfront-end to back-end

datapipelineUI componentMPEG-2 decoderAES decryptor155Live Objects Applicationsside by sidenestedfront-end to back-end

datapipelineUI componentMPEG-2 decoderAES decryptorreliable transport156Live Objects Applicationsside by sidenestedfront-end to back-end

datapipelineUI componentMPEG-2 decoderAES decryptorreliable transportunreliable transport157

side by sidenestedfront-end to back-end

datapipeline158

side by sidenestedfront-end to back-end

159

side by sidenestedfront-end to back-end

160

side by sidenestedfront-end to back-end

161

side by sidenestedfront-end to back-end

162

side by sidenestedfront-end to back-end

163

Live Objects Applications

UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy164

Live Objects Applications

UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy165

Live Objects Applications

UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy166

Live Objects Applications

UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy167

Live Objects Applications

UI componentMPEG-2 decoderAES decryptorreliable multicastIP multicastrecoveryprotocolhole-punchingUDP transportTCP transportmembership clientweb service proxy168

Live Objects Applications

datapipeline169

Live Objects Applications

datapipeline

datapipeline170

Live Objects Applications

datapipeline

datapipelinedatapipeline171

Live Objects Applications

XML172References

XMLactivatechannelreferencechannelchannelproxy173References

XMLactivatechannelreferencechannelchannelproxy174References

XMLactivatechannelreferencechannelchannelproxy175References

XMLchannelreferencechannelchannelproxypoints to176References

XMLchannelreferencechannelchannelproxypoints tochannelcontentXMLXML177References

XMLpoints to

points toXMLXML

178References

XMLpoints to

points toXMLXML

179References

XMLXMLXMLXML

XML

XMLXMLXML

XML

XMLXML

XML

XMLXML

XML

XMLXML180Ordinary Web Applications

side by sidenestedXML markup181

Live Objects Applications

XMLXMLXML

XMLXMLXML

XMLXML

182

Live Objects Applications183

Live Objects Applications

space184

Live Objects Applications

space

plane 1XML185

Live Objects Applications

space

plane 1

plane 2XMLXML186

Live Objects Applications

space

plane 1

plane 2

building 1XMLXMLXML187

Live Objects Applications

space

plane 1

plane 2

building 1

mapXMLXMLXMLXML188

Live Objects Applications

tile 1XML189

Live Objects Applications

tile 1XML

tile 2XML190Live Objects Applications

191Live Objects Applications

desktop192Live Objects Applications

desktop

text 2XML193Live Objects Applications

desktop

text 2XML

image 1XML194

Live Objects Applications

desktop

text 2XML

image 1XMLdesktop 2XML195

Live Objects Applications

desktop

text 2XML

image 1XMLdesktop 2XML

text 3XML196

Live Objects Applications

desktop 1197

Live Objects Applications

desktop 1XML

desktop 2198

Live Objects Applications

desktop 1XML

desktop 2

desktop 3XML199

Live Objects Applications

desktop 1XML

desktop 2

desktop 3XML

text 1XMLXMLXML200

Live Objects Applications

desktop 1XML

desktop 2

desktop 3XML

text 1XMLXMLXML201Conclusions202Cloud and edge technologies can work togetherbut we need a new storage abstraction.Checkpointed Channels (CCs)Support a variety of protocols: client-server, P2P, distributed replication, append log files, etc.,Can be used at any level of the protocol stack, at the fronted and at the backend, Can efficiently store structured mashup content,Can be hyperlinked into webs of CCs; these could serve as a basis for a new Web architecture.Conclusions203Thanks204