architecting cloud

Download Architecting cloud

Post on 03-Sep-2014

1.358 views

Category:

Technology

1 download

Embed Size (px)

DESCRIPTION

Architecting your application for the cloud

TRANSCRIPT

  • Architecting your application for the cloud
  • Traditional solution
    • Buy servers
    • Buy storage
    • Sign a CDN contract (Content Delivery Network)
    • Launch website/application
    • Manage scaling and provisioning
  • Cloud solution
    • Benefits from Cloud Computing:
    • No need to buy IT Infrastructure
    • Deploy worldwide
    • Scale up/down when needed
    • Save time
    • Focus on your business
  • Stage One The Beginning
    • Simple architecture
    • Low complexity and overhead means quick development and lots of features, fast.
    • No redundancy, low operational costs great for startups.
  • Stage 2 - More of the same, just bigger
    • Business is becoming successful risk tolerance low.
    • Add redundant firewalls, load balancers.
    • Add more web servers for high performance.
    • Scale up the database.
    • Add database redundancy.
    • Still simple .
  • Stage 3 The pain begins.
    • Publicity hits.
    • Squid or varnish reverse proxy or high end load balancers.
    • Add even more web servers. Managing contents becomes painful.
    • Single database cant cut it anymore. Splits read and write. All writes go to a single master server with read only slaves.
    • May require some re-coding of the apps.
  • Stage 4 The pain intensifies
    • Replication doesnt work for everything. Single writes database Too many writes Replication takes too long.
    • Database partitioning starts to make sense. Certain features get their own database.
    • Shared storage makes sense for contents.
    • Requires significant re-architecting of the app and DB.
  • Stage 5 This Really Hurts !!
    • Panic sets in. Re-thinking entire application. Now we want to go for scale?
    • Cant just partition on features what else can we use? Geography, lastname, user Id etc. Create user-cluster.
    • All features available on each cluster.
    • Use a hashing scheme or master DB for locating which user belongs to which cluster.
  • Stage 6 Getting a little less painful
    • Scalable application and database architecture.
    • Acceptable performance.
    • Starting to add new features again.
    • Optimizing some of the code.
    • Still growing, but manageable.
  • Stage 7 entering the unknown...
    • Where are the remaining bottleneck?
      • Power, Space
      • Bandwidth, CDN, Hosting provider big enough?
      • Firewall, load balancer bottlenecks?
      • Storage
      • Database technology limits key/value store anyone?
  • Amazon Services used Servers: Amazon EC2 Storage: Amazon S3 Database: Amazon RDS Content Delivery: Amazon CloudFront Extra: Autoscaling, Elastic Load Balancing
  • What is in step 1 Launched a Linux server (EC2) Installed a web server Downloaded the website Opened the website Now, our traffic goes up...
  • To reach fans worldwide, we need a CDN.
  • Changes in HTML code images/stirling1.jpg Becomes d135c2250.cloudfront.net/stirling1.jpg
  • What is in step 2 Uploaded files to Amazon S3 Enabled a Cloudfront Distribution Updated our picture location
  • Our IT Architecture needs an update
  • What is in step 3 We added Autoscaling, and watched it grow the number of servers We added Elastic Load Balancer
  • What we is in step 4 Launched a Database Instance Pointed the web servers to RDS Created a Read Replica Created a Snapshot
  • What is difficult about Databases?
  • Availablity Patterns
    • Fail-over IP
    • Replication
      • Master-slave
      • Master-master
      • Tree replication
      • Buddy replication
  • Master-Slave Replication
  • Master-Slave Replication
    • Assume both Master and Slave is running on Ubuntu Natty(11.04) with MySQL installed.
    • Configure the Master: we must configure the mysql to listen to all IP addresses. We move to
    • /etc/mysql/my.cnf
    • #skip-networking
    • #bind-address = 127.0.0.1
    • Set the mysql log file, the database name that we will replicate and tell that this will be the master
    • log-bin = /var/log/mysql/mysql-bin.log binlog-do-db=exampledb server-id=1
    • Then we made a restart:
    • /etc/init.d/mysql restart
  • Master Slave Replication
    • Now we enter the mysql on master server:
    • mysql -u root -p Enter password:
    • We grant all privileges for slave for this database:
    • GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY ''; FLUSH PRIVILEGES;
    • Then we run the following commands:
    • USE exampledb; FLUSH TABLES WITH READ LOCK;
    • This will show the master log file name and the read position: SHOW MASTER STATUS;
  • Master-Slave Replication
    • We make a dump of the database of the master server:
    • mysqldump -u root -p --opt exampledb > exampledb.sql
    • Or we can run this command on the slave to fetch the data from master:
    • LOAD DATA FROM MASTER;
    • Now we will unlock the tables:
    • mysql -u root -p Enter password: UNLOCK TABLES; quit;
  • Master-Slave Replication : Configure the Slave
    • First we enter the slave mysql and create the database:
    • mysql -u root -p Enter password: CREATE DATABASE exampledb; quit;
    • We import the database using the mysql dump:
    • mysql -u root -p exampledb < /path/to/exampledb.sql
    • Now we will configure the slave server:
    • /etc/mysql/my.cnf
    • We write the below information:
    • server-id=2 master-host=192.168.0.100 master-user=slave_user master-password=secret master-connect-retry=60 replicate-do-db=exampledb
    • Then we restart mysql:
    • /etc/init.d/mysql restart
  • Master-Slave Replication: Configure the Slave
    • We can also load the database using the below command:
    • mysql -u root -p Enter password: LOAD DATA FROM MASTER; quit;
    • Then we stop the slave server.
    • mysql -u root -p Enter password: SLAVE STOP;
    • And we run the below command to adjust the master informations:
    • CHANGE MASTER TO MASTER_HOST='192.168.0.100', MASTER_USER='slave_user', MASTER_PASSWORD='', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=183;
    • And then we start the slave server:
    • START SLAVE; quit;
  • Master-Master Replication:
  • Master-Master Replication: master1 configuration
    • we will call system 1 as master1 and slave2 and system2 as master2 and slave 1.
    • We go to the master mysql configuration file:
    • /etc/mysql/my.cnf.
    • Then we add the below code block. We show the path and socket path, the log file for the db to replicate.
    • [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock old_passwords=1 log-bin binlog-do-db=
    • binlog-ignore-db=mysql binlog-ignore-db=test server-id=1 [mysql.server] user=mysql basedir=/var/lib [mysqld_safe] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
    • mysql> grant replication slave on *.* to 'replication'@192.168.16.5 identified by 'slave';
  • Master-Master Replication: slave2 configuration
    • Now we enter the slave2 mysql configuration file.
    • [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock old_passwords=1
    • server-id