reducing the operations burden and increasing qos wso2 platform management and administration...
TRANSCRIPT
![Page 1: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/1.jpg)
Reducing the Operations Burden andIncreasing QoS with
WSO2 Platform Managementand Administration Tactics
Sanjaya RatnaweeraAmani Soysa
![Page 2: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/2.jpg)
Agenda
● Common Deployment Patterns● Deployment Synchronization● Registry Mounting● Setting up a worker-manager separated Appserver cluster -
Demo/Hands-on● Deployment Automation● Server Monitoring● Production Deployment Guidelines● Common issues in Production Setups
![Page 3: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/3.jpg)
WSO2 Product deployment patterns
● Standalone
● Clustered
● Cloud based
![Page 4: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/4.jpg)
Standalone Deployment
● User engages with one or many independent nodes
![Page 5: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/5.jpg)
Clustered Deployment
● User engages with the load balancer● Highly available● Concept of worker and manager nodes
![Page 6: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/6.jpg)
Worker-Manager Separation ?
➔ Proper separation of concerns◆ worker node - can ONLY deploy artifacts and read
configurations◆ manager node - authorized to add new artifacts and
make configuration changes➔ Lower memory foot in the worker nodes
◆ lesser OSGi bundles➔ Improved security
◆ management nodes can be behind the internal firewall & be exposed to clients running within the organization only, while worker nodes can be exposed to external clients.
![Page 7: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/7.jpg)
Cloud Based Deployment
● Extension of the Clustered deployment
● Highly scalable
● Span across multiple IaaSs
![Page 8: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/8.jpg)
Why Deployment synchronization ?
• Artifact distribution and deployment should be transparent to the end-users
• Manual artifact copying is acceptable for some extent in standalone product clusters.
• Will that work in an elastically scaling cloud environment ??
![Page 9: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/9.jpg)
Deployment Synchronization
● Distribute deployment artifacts and related meta-data across the cluster
● Make the cluster nodes consistent● Automated synchronization based on SVN ● Management nodes commits and worker nodes check-out
![Page 10: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/10.jpg)
![Page 11: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/11.jpg)
Deployment Synchronization<--Manager Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>true</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>
<-- Worker Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>false</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>
![Page 12: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/12.jpg)
Registry Mounting
● An approach to share registry space across cluster nodes● Local Data repository
○ Should not be shared● Configuration registry
○ Shared across multiple nodes of the same product● Governance registry
○ Shared across product platform
![Page 13: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/13.jpg)
Registry Mounting
![Page 14: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/14.jpg)
Worker-Manager Separated Cluster - Demo/Hands-On
● Setting up user management and registry databases● Configuring the Elastic loadbalancer● Configuring WSO2 AS as a management node● Configuring WSO2 AS as a worker node
![Page 15: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/15.jpg)
![Page 16: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/16.jpg)
Deployment Automation
![Page 17: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/17.jpg)
Configuration Management
![Page 18: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/18.jpg)
No matter how small you are
adapt yourself to configuration management
![Page 19: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/19.jpg)
Abstracted Configurationnode001 { appserver { version => "5.2.1", target => "/opt/appserver", cluster_domain => "mgt", }}
node002 { elb { version => "2.1.0", offset => "0", target => "/opt/elb", user => "root", } }
![Page 20: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/20.jpg)
Patch Management
![Page 21: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/21.jpg)
Orchestration
![Page 22: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/22.jpg)
Powered by Puppet
![Page 23: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/23.jpg)
![Page 24: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/24.jpg)
https://github.com/wso2/Puppet-Modules
![Page 25: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/25.jpg)
Server and Health Monitoring
● JMX based monitoring○ JConsole, Graphite
● Message Tracing Tools○ Wireshark, tcpmon
● System monitoring tools○ Ganglia, Cacti, Nagios
● Alerting○ Email, SMS, Phone-call alerts
![Page 26: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/26.jpg)
Production Deployment Guidelines
● Changing the default administrator credentials● Replacing SSL certificates/keys● Hardening the Operating System● Turning off unused services/ports● Switching the default H2 registry database to a production
ready database● Using secure vault to encrypt passwords in configuration files● Tune database connection pools
![Page 27: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/27.jpg)
Production Deployment Guidelines contd..
● Disabling HTTP access logs● Tuning heap memory● Tuning GC● Running servers as background processes
○ nohup, YAJSW
![Page 28: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/28.jpg)
Patching
● Follow the recommended procedure● Automate the patch applying process● Test the patch in dev and staging environments
○ Automated process is a must● Update production servers and restart in a RR manner● Maintain a log for patches
![Page 29: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/29.jpg)
![Page 30: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/30.jpg)
Common errors in production
● IO errors - Connection reset by peer, Broken pipe, Too many open files, I/O reactor has been shut down
● Database connection pool errors● OutOfMemoryErrors
![Page 31: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/31.jpg)
Production Troubleshooting Tips
● carbondump○ WSO2_HOME/bin/carbondump.{sh,bat}
● Replicate the issue in staging● Report JIRAs with detailed information
○ Upload carbondump output to the provided FTP server○ Attach the relevant deployment artifacts and logs
![Page 32: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/32.jpg)
Sanjaya Ratnaweera
Blog: http://samudura.org
Follow: @sanjayar
Amani Soysa
Blog: http://sparkletechthoughts.blogspot.com/
Follow: @poohdedoo
![Page 33: Reducing the Operations Burden and Increasing QoS WSO2 Platform Management and Administration Tactics](https://reader031.vdocuments.net/reader031/viewer/2022030304/5879aa7c1a28ab082c8b7bd3/html5/thumbnails/33.jpg)
Thank You