designing role-based database systems to achieve unlimited database scalability
TRANSCRIPT
2
Designing Role-Based Database Systems to Achieve Unlimited Database Scalability
Fernando G. GuerreroCEOSolid Quality Mentors
Session Code: DAT451
5
What forced me to prepare this session
The trigger
6
The "unpredictable workload" case
Millions of copies of a win app around the worldAll of them waiting for file safety, before using them
At undefined timesUnlimited number of potential files to checkUnpredictable level of workload at any date and timeUnpredictable geographical distribution of workload
Potentially, millions of requests per hourCan we solve it?
7
The "unreasonable scalability" case
The system already reached its limitsThey need to support workload 50 times higher
Scaling up is not an optionScaling out is not a simple task
They need to support over 15K transactions per second, with textual based data
Can we solve it?
8
Fernando G. GuerreroCEO of Solid Quality Global LLC, USAPresident of the SolidQ subsidiaries around the [email protected]
Microsoft Regional Director for Spain since 2004SQL Server MVP from year 2000 till 2007Usual suspect at many international conferences
Introductions
9
The SolidQ recipe
4 SQL Server MVPs old friends speaking at a conference in Palm SpringsA pleasant dinner at a nice restaurant2 bottles of my favorite Spanish wine1 bottle of a great single Malt WhiskyLots of honest and trustful work for 6 years
10
What are you going to see and hear today
This session
11
This session is about…
Lesson learned from “unreasonable” and amazing real systems from real enterprise customersBased on actual work performed by our Solid Quality mentorsLearning to think out of the box when designing DB systemsThinking first, acting later
12
This session is NOT about…
CodeDemosObscure trace flags and registry hacksAny particular commercial product or service offeringDescription about standard scalability techniques
13
First things First
14
First… Listen
15
Then, try to understand what their needs really are…
And what they really want
16
A database system is just one more tool to help achieving
business goals
17
Your technical common sense is your most powerful tool
18
If you don’t need to buy anything to implemnent a
solution, it does not mean that this is the cheapest solution
19
Role based DB system design
A single box might be sufficientDifferent tasks might imply different rolesRoles might be incompatibleYou’ll never know until you measure it
20
The unpredictable workload case
The Problem
21
Role based DB system design
A single box might be sufficientDifferent tasks might imply different rolesRoles might be incompatibleYou’ll never know until you measure it
22
Massive attack
23
The unpredictable workload case
The Solution
24
A common sense powered queue
Multi-layered server roles:The ReaderThe WriterThe Master Root
“Magic array” of virtual server namesSelf updated definitionsProprietary advanced hash-based checkingLocal vs remote storage
25
The Magic Array
26
The Reader role
Organized as a Readers tree with a Reader Root at the topVerifies signature of requested fileReplies to client with simple response:
SafeUnsafeUnknown
If unknown, checks parent Reader for further informationReceives updates from parent Reader
27
The Writer role
Works as a Reader as wellOrganized as a flexible array of WritersSends signature to the Master Root
and sends file asynchronously if necessaryReplies immediately to client with simple response:
SafeUnsafeUnknown
Receives updates from Distributor
28
The Master Root
Stores new files and signaturesCalls the qualification engine to qualify new filesDistributes updates to root ReaderDistributes updates to array of Writers
29
Massive dynamic deployment
30
The unreasonable scalability case
The problem
31
Scalability needs Jan 2005
10
10090 90
300 300
Workload vs CapacityWorkload Initial C Improved C Series4
32
Scalability needs Aug 2006
10
200
300
90 90 90
300 300
600
Workload vs CapacityWorkload Initial C Improved C
33
Side note: is there such thing as a write-
only transaction?or…
how much writing activity do you really expect?
34
Scalability needs Dec 2008
10 200 300 500
1500
3000
90 90 90 90 90 90600 600 600 600
2000
4000
Workload vs CapacityWorkload Initial CImproved C Role Based C
35
Scalability needs beyond 2009
10 200 3005001500
3000
6000
12000
90 90 90 90 90 90 90 90600600 600 600 600 6002000
4000
8000
16000
Workload vs CapacityWorkload Initial CImproved C Role Based C
36
The unreasonable scalability case
The solution
37
Role-based architecture
38
Detailed role assignment
39
Technology alone cannot beat what an experienced engineer can design on the back of an
envelope
40
Q & A
www.microsoft.com/teched Tech·Talks Tech·Ed BloggersLive Simulcasts Virtual Labs
http://microsoft.com/technet
Evaluation licenses, pre-released products, and MORE!
Resources for IT Professionals
Related Content
Breakout Sessions (session codes and titles)
Interactive Theater Sessions (session codes and titles)
Hands-on Labs (session codes and titles)
Hands-on Labs (session codes and titles)
Track Resources
Resource 1
Resource 2
Resource 3
Resource 4
1 Year Subscription!
Complete anevaluation onCommNet andenter to win!
46
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED
OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.