Total Cost of Ownership Zsolt Fabian Database Engineering

2  

1.  Who  is  Spil  Games?  2.  What  is  TCO?  3.  What  drives  the  costs?  4.  Possible  improvements?  

Outline of the talk

Who are we? Who  is  Spil  Games?    

4  

•  Company  founded  in  2001  •  350+  employees  world  wide  •  200M+  unique  visitors  per  month  •  45  portals  in  19  languages  

•  Casual  games  •  Social  games  •  Real  Kme  mulKplayer  games  •  Mobile  games  

•  35+  MySQL  clusters  

Facts

TCO Total  Cost  of  Ownership    

6  

•  1980  –  “Life  Cycle  CosKng”  •  1987  –  Gartner  Group  •  Total  Cost  of  Ownership  •  Analysis  of  enterprise  spending,  vendor  evaluaKon  •  Many  inter-­‐related  direct  and  indirect  costs  •  Problems  with  TCO  

What is TCO and why it is important?

CAPEX  –  Capital  Expenses   OPEX  –  OperaKng  Expenses  

Relevant Costs

7  

Infrastructure life-cycle

8  

•  CollecKng  data  •  Different  base  systems  •  4,3  TB  of  RAM  •  402  drives  wriKng  3  ExaBytes  •  3  DBA  •  12,3  kW  power  consumpKon  •  Asset  value  exceeds  €  500,000  •  Cables  are  available  in  all  colors  of  the  spectrum  J    

A quick inventory

9  

Aging servers

•  Component  failures  •  Power  consumpKon  •  Volume  of  deployment  

10  

•  The  most  obvious  •  Base  server  +  CPU  +  Mem  +  Disk  +  Controller  •  Replacement  parts  and  Warranty  •  NPV  –  Net  Present  Value  •  Mining  invoices  

CAPEX: Purchase costs

AFR  –  Annual  Failure  Rate   ALE  –  Annual  Loss  Expectancy  

11  

•  Business  need  •  Cost  of  downKme  •  Levels  of  HA  and  Fault  Tolerance  

•  Server  downKme  •  CPU  failure  •  MySQLd  crash  •  Solar  acKvity  

•  Combined  availability  formula:  

CAPEX: Purchase costs: HA

A  =  1  –  (1  –  Ax)  ^  n  

12  

•  Giving  back  J  •  MySQL  Standard  EdiKon:  $  2,000  

•  Enterprise:  $  5,000  •  Cluster  Carrier  Grade:  $  10,000  

•  Microsoj  SQL  Server:  $  6k+  /  core    •  Oracle  10/11g:  $  40k+  •  AddiKonal:  Technology  extensions,  Dashboards  

CAPEX: License costs

13  

•  Work  hours  spent…  •  SLA,  Responsibility,  Stress,  Insider  Threat  •  MySQL  annual  subscripKon:  $  2,000  •  Specialist  support  from  $  1,500  /  server  /  year  

•  Unlimited  servers  for  $  30,000  /  year  •  Depends  on  complexity  

•  €  2,700  /  server  /  year    

OPEX: MySQL Professional support

It's  cheaper  to  add  more  servers  than  hire  more  engineers.  

14  

•  Cage  –  Rack  –  Server  

•  Networking  •  Deployment  (+  fuel  +  Kme)  •  Power  density  constraints  

•  Let’s  say  €  60  /  server  /  month  

OPEX: Hosting: Colocation

15  

•  Wao  =  Ampere  *  Volt  •  €  0.2  /  kWh  

•  12,31  kWh  (~  16%  of  total)  •  €  21,300  annually  

•  Measurements  •  IPMI  •  Power  meter  J  

•  Contributors  •  HDD  (25%)  •  Roles  (5-­‐10W)  

OPEX: Hosting: Power

16  

OPEX: Hosting: Power

Putting it together

Pu@ng  it  together     The  model    

Cost  Item   CAPEX   OPEX   TCO  5y  

Base  System   ?  

Extra  memory   ?  

Extra  disks   ?  

Licence  fees   ?  

Replacement  parts  /  Warranty  

?   ?  

Co-­‐locaKon   ?  

Power   ?  

Professional  support   ?  

TOTAL   ?   ?   ?  

Pu@ng  it  together     Cluster  of  6  nodes  

Cost  Item  (6x)   CAPEX   OPEX   TCO  5y  

Purchase:  HP  DL380  G7   €  25,000  

Purchase:  Extra  memory     €  6,000  

Professional  support   €  16,200  

ColocaKon  cost   €  4,320  

Power  costs  @200W   €  2,100  

Replacement  parts    

€  1,500  

TOTAL   €  32,500   €  22,620   €  145,600  

Pu@ng  it  together     Cluster  of  6  nodes  

Purchase  costs  18%  

Professional  support  58%  

ColocaKon  15%  

Power  8%  

Replacement  parts  1%  

TCO 5 years

Improvements “Insanity:  doing  the  same  thing  over  and  over  again  and      expecSng  different  results”  -­‐  Albert  Einstein    

22  

Purchase prioritization

Extend  warranty  

MTBF  

Obsolete,  Not  Urgent  

Limited  reparability  

Repairable  

New  

23  

•  SSD  reduces  the  cost  of  IOPS  •  SSD  is  fast  for  reads,  but  not  for  writes  •  Has  limited  lifespan  •  NAND  Flash  and  RAM  based  SSDs  •  MLC  (MulK  Level  Cell)  vs.  SLC  (Single  Level  Cell)  •  R/W  in  pages,  but  always  erased  in  blocks  •  The  main  advantage  that  it  doesn’t  have  to  spin  

HDD vs. SSD: Inner workings

24  

•  Dropping  over  Kme  •  Facts  

•  Tsunami  •  Rare-­‐earth  minerals  

•  Cost  per  GB  is  cheaper  at  HDD  •  Some  predicts  that  in  5  to  10  years,  SSD  will  replace  HDD  completely  

HDD vs. SSD: Pricing

25  

HDD vs. SSD: Power consumption Type   Model   Idle  Power   AcSve  Power   Annual  cost  

contribuSon  

SSD   Intel  520   600  mW   850  mW   €  1,49  

SSD   Intel  X25-­‐M   60  mW   150  mW   €  0,26  

SSD   Intel  X25-­‐E   60  mW   4,2  W   €  7,35  

HDD   ST9146852S   5  W   7  W   €  12,2  

26  

•  MTBF:  ~  200  years  •  HDD  

•  AFR:  0,55%    (-­‐  1,5%  -­‐  2%  -­‐  …)  •  ALE!  

•  SSD  have  LIMITED  write  cycles  •  SLC:  100,000  writes  /  block  •  MLC:  10,000  writes  /  block  •  Typical  MLC:  5,000  writes  /  block  •  Future  developments  

HDD vs. SSD: Endurance

27  

•  Intel  520  SSD:  5  years  (with  normal  desktop  usage)  •  35  TB  write  endurance  (TBW)  •  Official  endurance  is  overprovisioned  (~  50  %)  •  SaturaKon  

•  Filesystem  overhead  (0.25  btrfs-­‐>  0.015  (ext4))  •  Readout  

•  S.M.A.R.T.:  Percentage  of  rated  lifeKme  used  •  UKliKes  •  RAID  controllers  

HDD vs. SSD: Endurance

28  

•  Determined  during  the  design  •  The  equaKon  is  about:  Buffers  x  ConnecKons  •  DIMM  populaKon  guide  •  DRAM  market:  OVERSUPPLY  

Memory configuration

29  

•  Extending  Linux  block-­‐cache  with  SSD  •  Block-­‐cache  works  by  caching  accessed  blocks  •  Perfect  sizing  -­‐>  performance  similar  to  SSD  only  •  Failures  

• Writethrough          cache  ||  storage    • Writearound              –(w)–>  storage  –(r)–>  cache  –>  • Writeback                      –(w)–>  cache  –(w)–>  storage  –>  

Flashcache

30  

•  6  x  300  GB  RAID10  -­‐>  2  x  100  GB  SSD  +  2  x  1  TB  RAID1  •  ConsideraKons  

•  I/O  rates  • Write  savings  

•  Overprovisioning  •  ParKKoning  

•  How  long  to  cache  • What  to  cache  

•  Less  HDD  purchases,  More  SSD  purchases  (High  ALE)  

Flashcache

31  

•  4  nodes  of  6  x  146  GB  HDD  •  CAPEX:  €  4,700  •  OPEX:  €  293  •  TCO  5y:  €  6,165    

•  4  nodes  of  2  x  600  HDD  +  2  x  Intel  520  SSD,  30%  opt.  writes  •  Replacing  them  every  year  (TBWy:  100  TB)  •  CAPEX:  €  11,680  •  OPEX:  €  110  •  TCO:  €  12,230  

Flashcache: Case study

32  

•  TCO  is  a  very  handy  (someKmes  misleading)  tool  •  RCO  -­‐  Relevant  Cost  of  Ownership  •  CAPEX:OPEX  raKo  •  Giving  back  is  OPEX  •  Keep  eye  on  your  power  consumpKon  •  Buy  DRAM  •  Consider  ALE  when  you  switch  to  SSD  

Summary

Questions?

34  

•  PresentaKon  can  be  found  at:  hop://spil.com/tcomeetup2013      

•  You  can  contact  me  at:    zsolt.fabian@spilgames.com  

Thank you!