opencms days 2012 - the dispatch - running opencms 8 on amazon cloud services
DESCRIPTION
The Dispatch Printing Company is a leading regional media company in the USA, anchored by its flagship newspaper The Columbus Dispatch. Its Dispatch Broadcast Group owns and operates two TV stations, the WBNS radio station, the Ohio News Network radio service, and a 24-hour cable news channel. This session is a case study in migrating OpenCms sites, generating millions of daily page views, from a traditional data center to the Amazon Web Services platform. Through this migration there were many lessons learned about how to successfully use Amazon's cloud service offerings to improve OpenCms scalability and lower total costs to the business. An overview of select Amazon services and how they have been leveraged in a production OpenCms environment will be presented. We will talk about possible uses for a variety of Amazon services including: EC2 - Implementation strategy for running OpenCms on Amazon's Elastic Compute Cloud virtual hardware CloudWatch - Provide detailed visibility into the health of an OpenCms environment Simple Storage System - Work with OpenCms's export functionality to push exported files directly to Amazon's web accessible storage space CloudFront - Leverage the power of a content delivery network for your OpenCms environment We will discuss the effort prior to launch to convince the business that Amazon would be reliable, allow for a disaster recovery plan, be secure, and save the business money. We will provide tips on how we setup our infrastructure to alleviate the various concerns the business had. The first service leveraged was Amazon CloudWatch. This service can provide a detailed look at the health of the entire OpenCms infrastructure with little to no custom development effort. This includes the ability to quickly create alerts and notifications for when anything goes wrong in your environment. We also decided to leverage Amazon Relational Data Services. We will present the trade-offs in the decision to use a managed data layer and how we justified taking the managed database approach. Finally, we will briefly cover the other Amazon services that have been used as a part of our OpenCms deployment including ElastiCache, CloudFront, Simple Queue Service, Simple Email Service, SimpleDB, and Amazon S3.TRANSCRIPT
OpenCms: Amazon Web
Services (AWS)
The Presenters David Landreman
Web Services IT Manager
Email: [email protected]
Twitter: @GraphIt2000
LinkedIn: www.linkedin.com/in/davidlandreman
Andrew Roth
Senior Internet Development Engineer
Email: [email protected]
Twitter: @guitarplayinpro
LinkedIn: www.linkedin.com/in/rothandrew
Newspapers, weekly periodicals, TV stations, and radio
stations in Columbus, Ohio, USA
23 websites hosted on Amazon in OpenCMS 8
http://www.dispatch.com
http://www.10tv.com
Millions of pageviews daily
5 Years of content
700,000 HTML files
1.5 Million Resources (images, video metadata, etc.)
NginX Web Tier (4 Servers)
Tomcat 7 App Tier (4 Servers)
OpenCms Customizations
Over 40 custom modules developed
Advanced RSS Feeds
Video Support
XML Content Editor Changes
Content Relationship Engine (CRE)
Custom User Administration Tools
Solr Search Integration
Workplace UI Enhancements
Advanced Properties Dialog
Shortcuts Bar
Custom RSS Feed Module
More than content management for websites
Amazon Kindle
Native iPhone / Android Apps
Facebook Pages
Support for value lookup in nested resources
E.g. Story Resource includes VFS path to Author
and the RSS should include author’s twitter
handle
Filtering by content type
Easily Extendable
Media RSS for Video feeds
GeoRSS
Video Support
Video management completely inside
OpenCms for content editors
Admin tools
Uploading new videos
Viewing transcode status
Integration with RSS module for MRSS feeds
Automatic video thumbnail generation
Thumbnail added to VFS
Only video metadata is stored in OpenCms
Video files stored out to a CDN
Over 1 Terrabyte video content
Content Relationship Engine (CRE)
New user interface to support a large number
of categories
700+ categories for advertising purposes
Tagging support
List functionality to search based on tags and
categories
Developed prior to OpenCms 8
Solr Integration
All OpenCms servers do not need to maintain
independent lucene query
Added features for public facing search
Faceting
Stopwords
Synonyms
Demo
Cloud Migration
Project to upgrade from OpenCms 6 to
OpenCms 7 (later upgraded to OpenCms 8)
Original plan was to migrate hardware from
physical to VMWare at our thrid party data
center
2 months prior to completion the decision was
made to migrate to Amazon Web Services
(AWS)
Amazon Selection Factors
Team familiarity with AWS
Ease of beginning an engagement
No contract
Cost
Limited payment options
Large client base (Netflix, Instagram, …)
Large selection of services beyond virtual
computing resources
OpenCms on AWS
S3 / EC2 Admin Tools
CloudWatch & Simple Notification Service
Simple Email Service (SES)
Relational Database Service (RDS)
ElastiCache
S3 / EC2 Overview
What is S3 (Simple Storage Service)
File storage in the cloud
Our uses:
Backing up a SQL dump of our production
database
Temporary storage of video content during
transcoding process
Serving some static web content (CSS, JS, etc)
What is EC2 (Elastic Compute Cloud)?
Web service that provides resizable compute
capacity in the cloud. (ie. servers)
S3 / EC2 OpenCms Admin Tools S3 Admin Tools in OpenCms
Allows for browsing the contents of all our S3
buckets from within OpenCms
Can download content directly from admin tool
EC2 Admin Tools in OpenCms
Lists ALL servers in our environment including:
Server status (running, stopped, etc)
Public DNS entry
Instance ID
Security Group
Link directly to Tomcat admin portal if server is
running it
CloudWatch &
Simple Notification Service (SNS) CloudWatch provides monitoring for
AWS cloud resources and applications.
Default metrics for all EC2 Instances
e.g. CPU, memory, disk IO
1 minute measurement interval
Tracking custom metrics
OpenCms Publish Times, Java
Heap Usage, Database
Connections
Alert Thresholds (SNS)
Text message, email, JSON posts
Simple Email Service (SES) Managed service for bulk email sending
CloudWatch Integration
Monitor deliveries, bounces etc. from within AWS
console
Multiple endpoints
SMTP
Web Service Calls
Uses
Currently send all website registration emails through
SES
Goal is to replace Exact Target with SES for mass
emails (newsletters, etc) to reduce costs
Relational Database Service (RDS) Managed Relational Database (PaaS)
MySQL, Oracle, MSSQL
We use MySQL
Change to traditional database administration
No access to server console
No access to a true SA account
Active / Passive Failover Configuration
Easy server upgrades
Automatic backups
ElastiCache Managed Memcached Service
Web service that makes it easy to deploy, operate,
and scale an in-memory cache in the cloud.
Integration of ElastiCache metrics with CloudWatch
Hit rates, Eviction Rates etc.
Similar to Alkacon OCEE Accelerator cache
Memcached allows for one global cache copy as
opposed to one on each app server
Reduces hits to our database servers, faster page loads ,
increases ability to handle high traffic volumes, and gets
us closer to goal of autoscaling.
Demo
The Future
• Automatic Horizontal Scaling
• CloudFront as a Content Delivery
Network (CDN)
• CloudSearch to replace Solr Search
Auto Scaling Issues
with Current
Architecture
Auto Scaling Issues with
Current Architecture (cont.)
Why autoscale?
Increased scalability and reliability
Reduced costs
Issues with AutoScaling OpenCms
Sticky Sessions
In the cloud sticky sessions are a bad thing
Direct communication between web and
application servers
Application server synchronization
Direction communication across application
servers (Alkacon Cluster Module)
Automatic Horizontal Scaling -
Solutions Memcached session management with Tomcat
Implement SQS (Simple Queue Service) for
communication between the workplace and the
application servers for application server synchronization
Allow web and application tiers to expand / contract
independently of each other
Load balance traffic between web and application layers
Automatic Horizontal
Scaling - Solution
CloudFront as a
Content Distribution Network (CDN)
Currently use Limelight for video distribution
Moving videos to CloudFront will be a big cost
savings
Front static content from OpenCms to decrease
page load times
Images, CSS, JavaScript
CloudSearch
Fully managed search service (PaaS)
API similar to Solr API
Reduced features
Replace our Solr implementation which
removes the need to manage a separate
server dedicated to search
Improved Scalability / Autoscale
Questions?
David Landreman
Web Services IT Manager
Email: [email protected]
Twitter: @GraphIt2000
LinkedIn: www.linkedin.com/in/davidlandreman
Andrew Roth
Senior Internet Development Engineer
Email: [email protected]
Twitter: @guitarplayinpro
LinkedIn: www.linkedin.com/in/rothandrew