1 a framework for highly available services based on group communication alan fekete idit keidar...
Post on 22-Dec-2015
224 views
TRANSCRIPT
1
A Framework for Highly Available Services
Based on Group Communication
Alan Fekete Idit KeidarUniversity of Sidney MIT
2
Highly Available Services
• Availability through replication
• Dynamic set of servers – For load-balancing, adding new servers– For fault-tolerance, when servers fail / detach
• Clients connect to ‘abstract’ service
• Preemptive migration: client can be migrated in on-going session / transaction
3
Inspired by
• Highly available Video-on-Demand (VoD)
[Anker, Keidar, Dolev ICDCS 99]
• Uses group communication
• Server written in 2500 C++ code lines, including all availability logic
• Use of group communication not obvious
4
Framework
• Servers store content units– Partially replicated among servers– Static: no updates to content
• Client-service interaction in sessions
• Service is stateful during session – Service stores changing context for client
• Content served according to client context
5
Examples
• VoD – Content unit = movie, partially replicated – Context: location in movie, transmission rate,...– Movie frames sent to client depend on context– Client can random access -- changes context
• Courseware
• Interactive queries
6
Design Goals
• Client request leads to appropriate response• Availability in face of failures
– Need replication (can be partial)– Need preemptive migration
• Availability for varying number of clients– Need to vary number of servers
• Simple clients, flexible service– Availability is servers’ responsibility
7
Possible Problems at Migration
• Lost request (sent to dead server) – “Stale” context– Irrelevant responses
• Lost response (sent by dying server)• Duplicate response
Study what failure patterns cause problems, costs of minimizing them
8
Our Solution: The Basics
• Framework, not service• Configurable in several parameters
– Support for different policies
• Primary server assigned to session• Preemptive migration to backup server• Backups mirror session context
– Context freshness at backup – configurable– 3 levels
9
Replication of Context Info:Three levels of Freshness
• Unit database per content unit – Replicated among all servers of content unit– Periodic updates, frequency configurable
• Context reflecting all client requests – at primary and 1st level backups
• Context reflecting server responses too – at primary
10
Group Communication (GC)
• Processes organized in groups, communication addressed to group
• Groups are dynamic (join, leave, crash,..)
• Groups can partition (“partitionable GC”)
GSend(G)
11
GC Service Interface
Input send( group, message )
Output receive( message )
Input join( group )
Input leave( group )
Output view( group, members, id )
view id is increasing
Multicast
Membership
12
Semantics: “Virtual Synchrony”[Birman et al. 87]
• Group members that remain connected see events in same order– events: messages, views (totally ordered mcast)
• Framework for “state-machine” replication with fault tolerance, local consistency– Connected members go through same states– New members get state transfer
• Use to replicated unit database
13
Highly-Available Service: Multicast Groups
ServiceGroup
ContentGroup
Crouching
ContentGroup
Gladiator
ContentGroup
Spy Kids
SessionGroup Session
Group
SessionGroup
15
Server Session Setup
• When start-session arrives, use local unit database to choose primary and 1st backups
• Primary and 1st level backups join session group (thus creating it)
• Primary sends session group name to client, serves content to client
16
Migration
• Triggered by view in content group • State transfer to new members• Use local unit database to choose new
primary and backups per client– Choose 1st level backup as primary if possible– By virtual synchrony, same decision made
• Chosen primaries and backups join session group, primary sends content, ...
17
Configurable Parameters
• Replicas per content unit
• 1st level backups
• Periodic updates frequency
18
Availability Analysis: Bad Scenarios
• Membership service not live, or does not give servers consistent views– Consistent migration decisions not made– Can lead to no service or duplicate service
• All content unit replicas crash – No service possible
• Context lost in migration– Risk depends on configurable parameters