real-world vpsx® server sizing · hp-ux 11.23 (11iv2) or higher ... (kernel 2.4.19 and libgcc...

EOM TechTip

of 15

VPSX Real-World Server Sizing October 2, 2015

Output Management

EOM TechTip

of 15

Abstract This document is a general discussion on the initial sizing of servers running the VPSX output management

server component of the LRS Enterprise Output Server. The various parameters and concerns that are

factors in VPSX server sizing are discussed. Several real-world examples taken from actual customer

production processing environments, including software/hardware specifications, processing volumes,

and system resource consumption are provided as a planning guide for the initial installation and

implementation of VPSX in a production environment.

This document is confidential and intended for use by LRS customers and shall not be disclosed or

delivered to any third-party(s) unless explicitly authorized by LRS.

Copyright 2014 Levi, Ray & Shoup, Inc. Rev. 10/02/2015 MAL Author: tr 4/7/2010

LRS, , VPS, VPSX, VPSX/OutputManager, and PageCenterX are registered trademarks of Levi, Ray & Shoup, Inc.

LRS and are service marks of Levi, Ray & Shoup. Inc.

All other brand and product names are trademarks or service marks of their respective holders

EOM TechTip

of 15

Executive Summary

All software products require and consume a certain amount of system resources, including, CPU,

memory, disk, and various other system resources that relate to networking and other I/O

functions. Software vendors typically provide a set of software and hardware requirements

(prerequisites). These specifications generally indicate what is required for the system to provide

base functionality in its most basic configuration. However, it is universally recognized that it is

unwise to rely on these minimum prerequisites, which rarely translate to acceptable performance

in production environments.

This often leads to the question: what are the “recommended” server specs for VPSX®?

Unfortunately, there is no one-size-fits-all answer to that question. What is best in one production

environment does not necessarily equate to what is best in another environment – that is, unless

the two environments are relatively similar (in terms of scope of usage and infrastructure makeup).

A number of variables (discussed in this document) must be considered.

The good news is that VPSX is an extremely efficient system capable of supporting high volumes

with excellent performance using relatively modest resources. Still, considering the resource

requirements for sizing a system relative to your specific environment is prudent.

It is important to understand that proof-of-concept testing (e.g., during a product trial) can usually

be accomplished adequately using modestly sized systems significantly smaller than would be

required to support a production environment. The production environment will require more

horsepower. A production environment will also likely require other practices not strictly necessary

in a test environment, such as clustering and other high-availability approaches, and other Best

Practices.

Understanding the variables involved with server sizing is one thing; being able concisely to assign

values to those variables is quite another matter that is not always easily determined. In some

cases, you simply have to begin with a best guess (“guestimate”) then employ standard capacity

planning practices and adjust as necessary. Virtualized environments make making necessary

resource adjustments relatively easy and painless.

There are many variables that factor into this question. This document will discuss those factors

that come into play when considering server sizing for a VPSX system. It will also provide a number

of examples of configurations used in VPSX production environments by some LRS customers.

Several of these examples are large enterprise environments that support huge volumes of output

across a large number of printers (in some cases, located all over the world) very efficiently.

EOM TechTip

of 15

Comparing these examples to your environment and extrapolating should provide a guideline of

where you should land.

Software Prerequisites

VPSX is supported on the following operating system platforms (for production environments):

Windows Server 2003/2008/2012 (32 and 64 bit versions) – or later

IBM AIX 5.2 or higher (32 and 64 bit versions)

HP-UX 11.23 (11iV2) or higher for PA-RISC or Itanium

Sun Solaris 5.8 or higher for Sparc (32 and 64 bit versions)

SuSe Linux for z/Series (Kernel 2.4.19 and libgcc 2.2.5-87 or higher)

SuSe and RedHat Linux for Intel (Kernel 2.4.19 and libgcc 3.2.2-22 or higher)(32 Bit)

SuSe and RedHat Linux for Intel (Kernel 2.6.16 and libgcc 4.1.2 or higher)(64 Bit)

VPSX can be run on Windows desktop operating system (Windows XP, Vista, etc.) platforms for

basic testing purposes, but is not recommended for production environments or

volume/performance testing scenarios. If VPSX is going to run on a 64-bit system, it is

recommended that the 64-bit version of VPSX be installed there.

A web server is required to provide access to the browser based VPSX administrator interface. The

LRS/Web Connect component will be installed on the web server (operating as a CGI program).

This can be a dedicated web server running on the same machine as the VPSX system or a shared

web server running elsewhere on the network (provided it is a supported web server platform).

Supported web server platforms include:

Microsoft Internet Information Server (IIS) version 6 or later

Apache

o Note: this must be the Apache distribution supplied with the operating system(s) listed

above. Apache distributions independently downloaded from the Internet are not

supported.

You should be aware that some web based applications on Windows that are using the same IIS

server as VPSX can cause conflicts that may interfere with LRS/Web Connect. These problems can

be difficult to troubleshoot. Microsoft Sharepoint is a known example of this, but there could be

others. Utilizing a dedicated IIS web server (e.g., when running VPSX on Windows Server) can help

avoid these conflicts. Adding IIS to a Windows system (running VSPX) is a trivial matter, and an easy

way to avoid potential problems.

EOM TechTip

of 15

LRS Enterprise Output Server Overview

The LRS Enterprise Output Server consists of three major components that provide an unprecedented level

of unique features and functionality not found in any solution from any other vendor as an integrated

enterprise level output management solution. Over 30 years of LRS Enterprise Output Management

experience have resulted in a solution whose scalability and performance is unmatched, and provides a

single enterprise output solution that extends across all applications and processing platforms.

The major components in the LRS Enterprise Output Management Server are:

Output Management Server: VPSX

Audit and tracking server: Innovate Audit

Document Management Server: PageCenterX

While the three components listed above represent a comprehensive enterprise level solution and are

typically implemented as a total solution, the individual components can be implemented individually.

The Output Management Server – VPSX

VPSX provides both generic and application specific interfaces for capturing output from applications on all

platforms. Output is processed as required, and delivered to an unlimited number of destinations of various

types (i.e., not limited to just printers). The output can (optionally) be retained in the VPSX spool for a short

period of time providing temporary storage for viewing, reprinting, etc. The web interface provides an

extremely powerful, yet easy to use, tool for help desk and operations personnel responsible for

troubleshooting and resolving problems, without the need to engage application or system administrator

resources.

The VPSX solution provides the flexibility to process any inbound datastream and ensure it conforms to

whatever format (any-to-any) is required by the destination. This functionality is available by tailoring the

solution to include specific datastream Transform products available from LRS. Advanced document

processing, document enhancement, and document re-purposing functionality is also available.

Audit Server – Innovate Audit

Innovate Audit collects accounting information from all LRS Output Management Server components in the

environment and provides access to critical data necessary for organizations to identify costs associated

with printing across the enterprise, and help identify key areas where cost savings can be (or have been)

realized.

The Document Repository Server – PageCenterX

EOM TechTip

of 15

PageCenterX provides the ability to store, manage, and logically organize documents of all types. Automatic

full document indexing enables extremely powerful search and retrieval capabilities for users who need to

quickly and easily locate documents containing specific information or data (e.g., a report containing data

for a specific patient or cost center). Documents are stored in customizable folder structures, and are

accessed by users through an intuitive easy-to-use web browser interface immediately familiar to any

Windows user. The powerful search functionality allows users to quickly locate documents located

anywhere in the repository (folder hierarchy) while preserving controlled access to important information

via flexible security controls.

The modular approach of the LRS Enterprise Output Server solution allows organizations to tailor the

solution to meet the specific requirements of the enterprise. This approach also allows organizations to

easily add functionality as future requirements demand.

VPSX Server Sizing Considerations

The purpose of this document is to provide guidelines for server sizing for the VPSX component, not specific

hardware recommendations. Due to the large number of variables involved, as well as various

“uncertainties” that are difficult to accurately determine prior to implementation, actual requirements may

vary when deployed in any customer production environment. Care should be taken to identify and value as

many of those variables as accurately as possible.

The following metrics are known to have an impact on server performance and capacity planning, as well as

the areas impacted by these items:

Parameter Most affected Lesser Affected

Total number of printers / destinations I/O Memory, CPU

Total number of Windows/Linux workstations connected to VPSX I/O CPU, Disk

Peak & Average Volume of Print Submission Requests I/O CPU

Total Number of Jobs/day and/or hour (jobs throughput) I/O Disk, CPU

Job Throughput Frequency (e.g., are many jobs generated by batch runs at specific hours; large percentage of total load bunched vs. evenly distributed throughout day?)

I/O CPU, Memory

Average Number of Pages Per Job (avg. size) Disk I/O

Number and Frequency of “Large” jobs Disk I/O, CPU

Amount of time jobs are ‘Retained’ (after successful print) Disk -

Total Number and Average size of ‘Retained jobs Disk -

Datastream Transforms utilization CPU Memory

EOM TechTip

of 15

Because there are so many variables, many of which often cannot be predetermined, as well as customer’s

hardware configuration standards and policies, LRS typically does not provide customers detailed

recommendations regarding production server configurations.

Notwithstanding specific recommendations, generally speaking, environments can be characterized as

“small,” “medium,” or “large.”

Smaller environments (typically meaning a few hundred printers or less with smaller volumes) will function

adequately on modestly sized servers (meaning entry-level server systems).

Medium sized systems typically include systems that range from 1,000 – 5,000 printers, processing moderate

amounts of data (e.g., <400,00-500,000 jobs/mo.).

Large systems may support many thousands of workstations, or large volumes of output from line-of-

business applications (e.g., ERP or EMR), and/or drive upwards of tens of thousands of printers. It may not

be unusual for throughput volumes in large system environments to exceed 2-3M jobs/mo. (or as high as 10

million or more pages/mo.).

OS System Settings & Tuning

In large enterprise implementations processing large volumes of output, consideration of various system

parameters (at the Operating System level) and the values of those settings would be prudent. These include

various parameter that affect TCP/IP stack performance, as well as items related to filesystem limits and

performance. The default settings (as well as the specific parameters themselves) vary between the various

OS’s (Windows vs. Linux vs. AIX vs. HPUX, etc.).

An example of this consideration is the ‘Extended Filesystem’ (Ext) level that being utilized in Linux. Ext was

originally created to extend the storage limits and provide other performance improvements in current Linux

distributions. The ‘Ext’ level affects system parameter such as, total filesystem size, number of extents

(related to blocking and fragmentation), pre-allocation, number of subdirectory limits, journaling, and

general filesystem performance. A specific example of this is that could affect VPSX environments is, Ext3

limits the number of subdirectories in a directory to 32,000. In VPSX, the ‘vpsxroot’ subdirectory under the

‘lrsroot’ directory contains subdirectories for each printer definition, as well as each printer has its own

subdirectory under the ‘spool’ directory. In large systems, it can become possible to exceed the 32,000

subdirectory limit. Converting the filesystem to Ext4 the maximum to 64,000 subdirectories, as well as allows

for huge maximums for individual files (up to 15TB) and the total filesystem size (up to 1 EX (Exabyte) – 1 EB

= 1024 PB (Petabytes), 1 PB = 1024 TB (Terabytes)). Ext4 also provides other performance and reliability

enhancements. As such, there is really no reason to not convert any medium/large Linux system supporting

VPSX to Ext4.

EOM TechTip

of 15

The above is provided only as an example and is not a comprehensive list of items that could (or should) be

changed when tuning an OS. As matter of course, it is recommended that any modification to TCP/IP tuning

parameters or filesystem or OS parameters be done only by a technician highly proficient in that OS and

familiar with the specific changes being made and how they affect the system overall (including any possible

liabilities or issues). The specific parameter and their values must be determined by the customer, as

appropriate for their environment (LRS makes only general recommendations, and does not typically specify

or recommend specific values).

Generally speaking, though, the OS should be tuned appropriately to the environment it is supporting.

EOM TechTip

of 15

Virtualized Environments & Clustering

The VPSX system will function in any virtualized environments (e.g., VMware) capable of running any of the

supported operating systems described at the beginning of this document. There are no known issues with

VPSX running in a virtualized environment.

Customers typically prefer to install on a virtualized environment due to the ease of adjusting the amount

of hardware resources available to the system as necessary to sustain a desired level of performance under

varying production loads.

In addition, nearly all customers deploy VPSX servers in a cluster or some high-availability solution. The VPSX

system is fully compatible with all common clustering solutions, but is not a “cluster aware” application.

VPSX is defined to the cluster as a generic cluster application. There are typically two VPSX nodes in a cluster

operating in active-passive mode.

The only specific requirement related to HA configurations is that the ‘LRSROOT’ filesystem can only be

accessed by one (active) system at a time, and it must be located on a shared disk resource (e.g., SAN, NAS,

etc.) accessible to both systems supporting VPSX (active and passive). Beyond this, the configuration of the

HA environment itself, as it relates to VPSX, is unique to the specific HA solution being utilized; LRS does not

provide specific recommendations on which HA solution to use or how to set it up.

With regards to resource allocation, again, we will refer to the “small,” “medium,” “large,” virtual machines,

which can be sized according to the following general guidelines (per virtual server).

Environ. Size CPU Cores Memory Disk

Small 1-2 4 GB 50GB or more

Medium 2-3 8-12GB 150GB or more

Large 4 12-16GB 500GB or more

Again, these are general guidelines; these are not guarantees of VPSX performance in your environment.

What is necessary to provide acceptable performance (which itself is subjective, based on the demands of

your business and your requirements) is unique to your specific environment. It would be prudent to apply

standard capacity planning practices to monitoring these systems on a regular basis and adjust system

resource allocations appropriately.

EOM TechTip

of 15

Real-World Examples

LRS customers will occasionally provide LRS information about their production implementations (server

configurations) and the performance they experienced in their environment. This section lists various

details about actual customer environments, the specifics of the hardware they are using in production,

and a general indication of system resource consumption and performance. These cases can be used

anecdotally to draw conclusions on hardware requirements in other production environments. The

industry segment for these accounts is noted in each case (which may be helpful when looking for

examples in you specific industry), but actual customer names are confidential and can only be disclosed

with the permission of that account.

Typically, VPSX is running on a dedicated server – no other applications are running on the server. This is

particularly true in higher volume environments. Whether or not you are using the LRS ‘Transform’

products to perform datastream conversion in VPSX has a direct impact on CPU and memory utilization.

In addition, the “Retention” policies (i.e., the length of time documents are retained in VPSX after being

successfully printed) can have affect VPSX startup time, as well as have a major impact on disk usage. It

should kept in mind that this feature is not intended to be used for long-term document storage (the LRS

PageCenterX product would be the correct solution for that).

In the examples below, average CPU utilization in the single digits by itself might be construed as machine

that is oversized. However, running CPUs at very high levels generally has a very undesirable effect on

overall system performance. Also, CPU utilization spikes can (or typically do) occur during high processing

times, such as nightly batch runs or various other types of circumstances.

Please note that the disk utilization metrics do not take into account space used for temporary working

files utilized during VPSX operation. As such, the statistics noted do not account for spikes in this metrics.

Exceeding (or coming close) to the total amount of disk allocated can have negative consequences.

These statistics generally represent average utilization. When considering these cases as guides for server

sizing, allowances must also be made for the CPU, memory, and disk specifications to allow for peak

processing times.

EOM TechTip

of 15

Case 1 – Global Chemical Producer and IT Outsourcer

Platform Number of Servers

CPU Utilization

Print Volume

Job Retention

Disk Utilization

IBM p570 LPAR 2 x virtual Power6 CPUs 8GB Memory

2 6.5% avg. 6.2M Jobs/mo. 14.3M pgs/mo.

1.2M jobs 205GB avg. (50GB retained jobs)

Additional Information:

AIX

1 Primary cluster

1 hot standby cluster (AIX)

Feeds to downstream VPSX systems at 15 remote customer sites

31 locations in 10 countries

14,000 defined printers

350 connected SAP systems

50 connected Windows systems (Oracle and UNIX apps)

Continuous operation – no scheduled downtime

Utilizing LRS Transforms to provide international language (UNICODE) support, as well as embedded barcodes.

All printers administered from centralized VPSX web interface.

Case 2 – Global Product Manufacturer


CPU Utilization

Print Volume

Job Retention

Disk Utilization

IBM 750-8223-E8B 6 x Power7 CPU 3.0 Ghz 24 GB Memory

2 60% avg. 20,000 jobs/hr. avg. ~480,000 jobs/day avg. 98,000 jobs/hr. peak

- 205GB avg. (50GB retained jobs)


AIX 6.1

Not virtualized

500 connected SAP systems

260,000 Windows desktop – Direct IP, VPSX providing Printer Driver Management

30,000 printers defined (60% Windows & SAP, 40% Windows only)

80% of volume processes by LRS Transforms (server mode)

Standard network laser printers, MFPs, plotters, and Zebra label printers

Replace +500 Windows print servers

750 different printer models

500 different printer drivers

“NOT VALID” watermark being printed on all output from any non-production SAP system

EOM TechTip

of 15

Case 3 – Large National Insurance Company


CPU Utilization

Print Volume

Job Retention

Disk Utilization

Virtual 2 Xeon L7555 1.87 GHz CPUs 4GB Memory allocation 60 GB Disk Allocated

4 6.5% avg. 6.2M Jobs/mo. 12.5M pgs/mo.

--- 50GB


Windows desktop printing

225,000 Windows workstations

9,000 defined printers each server (36K total)

Replaced 650 Windows Print Servers

Red Hat Enterprise Server Linux 6.4

4 servers deployed in 4 separate regional datacenters across US (for redundancy, disaster recovery purposes)

Virtualization by VMware

Case 4 – Medium Sized Healthcare System


CPU Utilization

Print Volume

Job Retention

Disk Utilization

HP Blade BL460 4GB Memory 4 Cores – 667 MHz

2* 10% avg. 650K Jobs/mo. - 8.7 GB


Windows desktop printing


Dedicated servers

* Clustered

Utilizing LRS Datastream Transforms

Two servers behind a load balancer (for HA only – no load balancing)

Jobs Retained 48 Hours

Cerner EMR

Case 5 – Large Sized Healthcare System


CPU Utilization

Print Volume

Job Retention

Disk Utilization

Dell Poweredge 2950 Intel Quad Core Xeon X5355 4 GB Memory

2* 15% avg. 1.3M Jobs/mo. --- 15 GB


Windows Server 2008


Epic

12 hours Retention (1 or 7 days on some queues)


* Clustered (Active / Passive)

EOM TechTip

of 15

Utilizing third “hot standby” ALT server – Dell X5310

Case 6 – Global Heavy Equipment Manufacturer


CPU Utilization

Print Volume

Job Retention

Disk Utilization

IBM Power 6 p570 LPAR 2 Quad Core Xeon CPUs (E5345) 4 GB Memory

1* 6.5% avg. 350K Jobs/mo. --- 40 GB


AIX

Virtualized into two virtual systems – IBM pSeries Hypervisor

SAP environment


No Retention


* Clustered (Active / Passive) - Veritas

Case 7 – Global Automobile Manufacturer


CPU Utilization

Print Volume

Job Retention

Disk Utilization

Dell Power Edge 720 24 Core CPU 128 GB Memory

1* 30-90% 4.5 M Jobs/mo. --- --


SuSe linux (SLES 11 SP3)

SAP environment

1,920 VPSX printer queues on primary server

Primary server also runs complex scripted workflows and Streamserve composition for all print files

Additional VPSX SuSe linux systems with 4 CPU and 8GB RAM running virtualized using Xen

EOM TechTip

of 15

Ongoing Monitoring and Capacity Planning

As with any significant systems, organizations will need to monitor the resource usage and employ standard

capacity planning practices. It is recommended that snapshots be monitored on a daily basis the first week

of production processing, then weekly for the next 4-5 weeks, and resource allocations adjusted as necessary

to meet processing requirements. After the first 5-6 weeks of production processing, capacity should be

monitored monthly on an ongoing basis.

VPSX systems administrations screens provide access to system statistics pertaining to performance and

resource utilization, which may be useful for monitoring and capacity planning. Information available via

these screens includes:

Release levels Pages printed

Current allocated memory Lines printed

Max allocated memory Bytes printed

Current active printers Number of queues

Highest active printers Files in Output Queue (current)

Current printing Files in Retain Queue

Highest printing Available Spool Space

Jobs printed

VPSX also provides a “Dashboard” feature that allows active monitoring of several of these key performance

indicators at a glance.

EOM TechTip

of 15