aai-3218 production deployment best practices for websphere liberty profile
TRANSCRIPT
© 2015 IBM Corporation
Production Deployment Best Practices for WebSphere Liberty Profile
Erin Schnabel WebSphere Liberty Profile Lead [email protected]
Christopher VignolaWebSphere System Management [email protected]
Topics
• Profile Configuration
• Topologies & Practices
• Application Deployment
• Management Configuration
• Numbers and Sizes
2
Profile Configuration
3
Configuring Data Sources
• Use top-level libraries
• Allows multiple data sources and applications to share the same class loader
<library id="DB2JCC4Lib">
<fileset dir="C:/DB2/java"
includes="db2jcc4.jar db2jcc_license_cisuz.jar"/>
</library>
<dataSource id="db2" jndiName="jdbc/sampleDB">
<jdbcDriver libraryRef="DB2JCC4Lib"/>
<properties.db2.jcc databaseName="SAMPLEDB"
serverName="localhost" portNumber="50000"/>
</dataSource>
<application location="myApp.ear">
<classloader commonLibraryRef="DB2JCC4Lib"/>
</application>
4
Configuring Data Sources: Connections
• Connection pooling and timeouts
• Reduce connectionTimeout to confirm that max pool size is
being exceeded
• Increase maxPoolSize if you see connection timeouts
• If you don’t need two phase commit:
• Use ConnectionPoolDataSource rather than XADataSource
• Data sources can enlist in a global transaction as a one-phase
resource even if they are not XA-capable.
5
Configuring Data Sources: Connection sharing
• Set isolation level property instead of programmatically
• Use isolationLevel property on datasource
• Set in resource reference binding/extension
• Declared isolation level allows for better matching/sharing of
connections
• Use containerAuthData instead of user/password
• Prevents applications using res-auth=Application from
accessing the data source with container credentials
6
Threading and Thread pools
• Auto-tuning thread pool
• Optimizes for executor throughput (measured every 1.5 sec)
• Dynamically adjusts between coreThreads and maxThreads
– Default coreThreads = (2*processorCores)
– Default maxThreads = MAX_INT
• Not usually necessary to tune the thread pool
• Some workloads may warrant increasing coreThreads
• Long-running / Outbound-to-self
• Measure performance before and after adjustment
7
Separate App and Admin HTTPS traffic
• Configure Virtual Hosts to isolate the application from other
(internal/administrative) traffic.
8
<httpEndpoint id=“appEndpoint” … />
<!– Restrict access to default_host: only accessible via default
endpoint -->
<virtualHost id=“default_host”
allowFromEndpointRef=“defaultHttpEndpoint”/>
<!-- define an application-specific virtual host -->
<virtualHost id=“applicationHost”
allowFromEndpointRef=“appEndpoint”>
<hostAlias>*:${app.http.port}</hostAlias>
<hostAlias>*:${app.https.port}</hostAlias>
</virtualHost>
<!-- configure plugin to route to the app-specific endpoint -->
<pluginConfiguration httpEndpointRef=“appEndpoint” />
server.xml:
<virtual-host name=”applicationHost" />
Ibm-web-bnd.xml
Virtual host
binding required in
the application
SSL Certificates and KeyStores
• Use only officially signed SSL certificates
• Use a separate keystore for inbound vs. outbound
• Encode or encrypt passwords in your configuration
<ssl id="defaultSSLConfig"
keyStoreRef="defaultKeyStore"
trustStoreRef="defaultTrustStore" />
<keyStore id="defaultKeyStore"
location="${server.config.dir}/key.jks"
type="JKS" password="{aes}..." />
<keyStore id="defaultKeyStore"
location="${server.config.dir}/trust.jks"
type="JKS" password="{aes}…" />
9
Security - General
• Harden your environment – extensive resources:
http://www.ibm.com/developerworks/websphere/techjournal/1210_lansche/1210_lansche.html
http://www.ibm.com/developerworks/websphere/techjournal/1303_lansche/1303_lansche.html
• Avoid vulnerabilities – keep service current.
• Register for support notifications at ibm.com/support .
10
Topologies& Practices
11
Topologies
• Standalone
• collective Assisted Lifecycle
• Collective
When to Use Which Topology?
• Standalone – small, isolated, or situational deployments
• collective Assisted Lifecycle
• Small/Med deployments
• Existing admin skill
• multi-server QoS
• Collective
• Small/Med/Large deployments
• Lightweight environment
• Multi-server QoS
13
Universal Practices
• Server Package Deployment
• High Availability – Rule of 3
• Automated deployment
• Secure environment
14
QoS Practices (Optional)
• Dynamic Routing
• Auto-scaling
• z/OS integration
15
Gold Standard – HA Collective
16
Collective Controller
Replica Set
CCCC
CC• Include AdminCenter
• Use Dynamic Routing
• Use Auto-scaling
Machine Boundary
AppServerAppServerLiberty
Profile
Per
App
IHSIHS
IHS
Per
Collective
z/OS Integration
• Use read-only mount point for servicing runtime
• Use PROCs for running servers
• Use START/MODIFY/STOP commands for server lifecycle
• Use SAF Registry support
• Exploit platform integration as necessary:
• zosWLM
• zosTransaction
• zosLocalAdapters
17
ApplicationDeployment
18
Build & Deploy
• Use server package for deployment – 3 models
• Developers direct deploy
• Developers hand off app, admin packages/deploys
• Developer direct deploy w/packaging automation to ensure
“approved server config”
>> Establish build/deploy pipeline
• Automate – use script, Chef, Urbancode, etc
19
Example Build/Deploy Pipeline
20
Jenkins
Liberty
Server Package Deployment: Managing Config
21
$WLP_USER_DIR/MyServer/
server.xml
apps/
MyApp.war
deploy (apply target overrides)
$WLP_USER_DIR/MyServer/
server.xml
configDropins/
overrides/
prod.xml
apps/
MyApp.war
Server Package Update
• Update through build process
• Blue/Green deploy
• Dual install locations
• Ripple start (stop old/start new, JVM by JVM)
• Delete old instance at your convenience
22
Upgrade – 1 of 5
23
host1.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
Upgrade – 2 of 5
24
host1.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
Upgrade – 3 of 5
25
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
Upgrade – 4 of 5
26
host1.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host2.com
http/s: 9080/9443
status: STOPPED
/wlp-blue/usr/servers/prod1
/wlp-blue/bin
/wlp-blue/lib
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
Upgrade – 5 of 5
27
host1.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
host2.com
http/s: 9080/9443
status: STARTED
/wlp-green/usr/servers/prod1
/wlp-green/bin
/wlp-green/lib
ManagementConfiguration
28
High Availability Services
• Collective scope
• Built-in to collective controller replica set
• Host scope (e.g. scalingMember-1.0 feature)
• Uses local port
29
Collective SSL
• Collective root, member root
• First controller establishes “true root”
• Must be copied to subsequent replicas
• Trust between
• Controllers and members
• Members and controllers
• Members on same host
• Certificates
• Signers
• Identity
30
Security
• Use external user registry – e.g.
• Ldap
• SAF
• Multiple registries allowed – e.g. as granular as per cluster.
• Collective Controller Replica set must use same registry.
31
Highly Available Collective Controller
• Three controllers minimum. Odd numbers only.
• Up to 2000 member per controller
Recommended: Max(Members/2000,3)
+1(for controller failure)
+1(for network partition)
• Use configDropIns directory – it’s replicated!
• Configure members with controller failover addresses.
32
CCCC
CC
Dynamic Routing
• Requires IHS (or Apache).
• Use dynamicRouting-1.0 feature in controllers.
• Double-layer IHS to simplify firewall management.
• Terminate SSL at earliest opportunity.
33
CCCC
CC
IHSIHS
IHS
Auto-scaling
• Put scalingController-1.0 feature in at least 3 controllers.
• Set hostSingleton port for vertical scaling.
• Set min instances based on average demand.
34
AppServerAppServer
AppServer
Elastic
Resources
CCCC
CC
Auto-scaling Policy
• Leave headroom on max settings (e.g.
• CPU <= 90%
• Heap <= 90%
• Memory <=90%
• Include scaling policy in server package
• scaling-metadata.xml
35
Use Admin-metadata
• Owner
• Contacts
• Note
• Tags
• admin-metadata.xml – part of server package
• assignable to host, server, application, cluster, runtime
• There is also an API
• Used in AdminCenter to search and set views
36
Numbers and Sizes
37
38
Liberty Profile Collective Size Design Considerations
• Collectives are design for large scale
• What limits the size of a WebSphere Collective?
• Breadth and currency for shared Information across controller JVMs
• Communication and coordination of shared information
• Product features scale differently with large collective size
• Most affected : collectiveController, scalingController
• It is possible to create a large topology collectiveBut will it function to your requirements?
• Collective works well with defaults settings. Some environments may
require tuning.
39
When do you need multiple collectives?
• Isolation
• Development vs testing vs production
• Critical applications
• Backup site
• Lines of business
• Each funding area may have different policies for when to apply
fixes, when to upgrade
• Geography:
• Controllers can span data centers with qualifying* network and config
• Members can span data centers with tuning
• Large collectives require planning to avoid “urban sprawl”
40
How Large a collective can I create?
• No hard limit – trust, but verify
• Lab tested:
• Tested 10,000 members
• 5 controllers
• 50 VMs
• ~200 members per VM
• Controller VMs: 20GB memory+6 CPUs
• Member VMs: 64GB+16 CPUs
41
How Many Host OS Instances per collective
• No Design Limit
• Typical large topology up to hundreds of hosts.
• Practical Limits
• Operations may take longer:
– Configurations, server deployment
• Notifications flowing back to collective controllers
• Load on controllers with concurrent operations
• 50 hosts per collective controller guideline
42
How Many Application Servers per Host OS?
• Keep WAS JVMs completely within physical memory
• Allow for overhead
• Process footprint about 1.25 to 1.5X maximum heap size for 32 bit
heaps
• 1.6x to 1.8x for 64-bit heaps
• If App Server gets swapped out, add more memory or
else move to different host OS.
• Ensure sufficient CPU is allocated
• Especially for hypervisor
• Avoid CPU starvation
43
How Many Applications Per Server
• Balance between resource usage and isolation
• With one application per server
• One bad application does not bring down all other applications
• Easier to tune each application in isolation
• With more than one application per server
• Less resources: cost of application server runtime amortized
across multiple applications
• Smaller topology to manage
• Configure as much isolation as you can afford
• If you have 300 applications clustered on 3 nodes
– Complete isolation 900 JVMs
– Complete sharing 3 JVMs
44
System Management Best Practices
• Set and track your performance goals• Measure performance of your most commonly used operations
• Track changes as you increase collective size
• Track changes over time to identify new issues
• Use Scripting (or your own Java framework)• Automatable, repeatable, testable
• Don’t overload collective controllers• Give enough memory + CPU
• Run AdminCenter in multiple controllers to spread load
• Don’t co-locate with resource intensive processes, e.g., application servers with heavy load
Wrap Up
Covered
• Topologies & Practices
• Profile Configuration
• Application Deployment
• Management Configuration
• Numbers and Sizes
46
Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the
agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.
Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank YouYour Feedback is
Important!
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.