performance assessment - content server 10

Product:

Version:

Task/Topic:

Audience:

Platform:

Document ID:

Updated:

OpenText Content Server

10.0.0

Performance

Administrators, Decision Makers

All

500113

October 7, 2011

White Paper

Content Server 10.0 – Performance Assessment Geoff Obbard, Scott Tindal, Senior Performance Analysts, Michelle Quesnel, Performance Analyst and Alan Robertson, Quality Assurance Analyst, OpenText Performance Engineering

TOGETHER, WE ARE THE CONTENT EXPERTS CHAMPION TOOLKIT

Document Title

TOGETHER, WE ARE THE CONTENT EXPERTS CHAMPION TOOLKIT 2

Contents Introduction ................................................................................................................. 4

Objectives ............................................................................................................... 4 Web Services Test Objectives ................................................................................... 5 Tests and Results ........................................................................................................ 6

CWS Tests .............................................................................................................. 6 1. Ingestion Isolated Operation Tests ..................................................................... 6 2. Ingestion Load Tests ............................................................................................ 7 3. Retrieval Isolated Operation Tests .................................................................... 13 4. Retrieval Load Tests ........................................................................................... 14

Web Services Test Summary ......................................................................... 19 Web User Interface Test Objectives ........................................................................ 20 Web User Interface Isolated Operation Test ........................................................... 21 1. Browse Requests ............................................................................................... 22 2. Create Requests ................................................................................................. 23 3. Fetch Requests ................................................................................................... 24 4. Personal Menu Requests ................................................................................... 26 5. Function Menu Requests ................................................................................... 27 6. Search Requests ................................................................................................ 27

Summary .............................................................................................................. 32 Web User Interface Load Tests ................................................................................ 33

Method .................................................................................................................. 33 Load Test Results ................................................................................................. 34

Load Size Summary ....................................................................................... 36 Throughput ..................................................................................................... 37 Execute and Response Times ....................................................................... 40 Resource Usage ............................................................................................ 44

Thread Usage .......................................................................................... 44 CPU Usage .............................................................................................. 47 Memory Usage ........................................................................................ 49

Load Test Summary ....................................................................................... 51 Stress Test Results ......................................................................................... 52 Stress Test – Content Server 9.7.1 ................................................................ 52

Throughput & Response Time ................................................................. 52 Resource Usage ...................................................................................... 53 Response Time Correlation ..................................................................... 55

Stress Test – Content Server 2010 ................................................................ 56 Throughput and Response Time ............................................................. 56 Resource Usage ...................................................................................... 56 Response Time Correlation – Content Server 2010 ............................... 58

Stress Test Summary ..................................................................................... 59 Conclusion ................................................................................................................. 61

Document Title


Appendix A: Test Environment Architecture .......................................................... 62 Appendix B: Test Environment Hardware .............................................................. 63 Appendix C: Non-Default Configuration ................................................................. 64 Appendix D: Data-Sets ............................................................................................. 66

Document Sizes Folder: ....................................................................................... 66 Folder Sizes Folders: ............................................................................................ 66 News Article Data-Set ........................................................................................... 67 Ingested Document Data Set ............................................................................... 68

About OpenText ........................................................................................................ 70

Document Title


Executive Summary This paper describes the testing efforts undertaken by OpenText to determine the key performance characteristics of Content Server 10.0 relative to those of Content Server 9.7.1. Tests are broken down into Content Web Services ingestion and retrieval tests, Web User Interface isolated operation, load, and stress tests.

Results of the Content Web Services ingestion load tests showed that even at eight threads, Content Server 10.0 achieves 30% higher ingestion throughput than 9.7.1, and a peak throughput of over 53 documents created per second at 40 threads. This is more than three times the peak ingestion throughput of Content Server 9.7.1.

Though Content Server 10.0 exhibits almost double the Content Web Services retrieval throughput of Content Server 9.7.1 while at the same number of threads (5), the most significant difference occurs at peak throughput, where Content Server 10.0 achieves over 4½ times the capacity of Content Server 9.7.1 (over 180 docs/second in 10.0, compared to less than 40 docs/second in 9.7.1).

Results of the load tests of Content Server 9.7.1 and 10.0 in nearly identical configurations show that Content Server 10.0 is able to deliver responses to a larger volume of requests faster than Content Server 9.7.1. Coupled with the increased scalability of threads, and address space available in a 64-bit process, a single instance of Content Server 10.0 should be able to handle a significantly higher volume of requests, and respond to them quicker, as compared to previous releases of Content Server.

Results of the stress test showed that the software bottle-necks that prevented Content Server 9.7.1 from using available hardware effectively have been removed from Content Server 10.0, and that a single instance of Content Server 10.0 is able to handle two and a half times the load of an instance of Content Server 9.7.1. By virtually eliminating the need to install multiple instances on a single server (vertical scaling), Content Server 10.0 significantly reduces the amount of effort needed to administer larger scale deployments.

For the most part, results of these tests showed either no loss in performance, or significant improvements to performance with Content Server 10.0. The most notable areas of improvement in Content Server 10.0 include:

1. Improved thread scaling 2. Improved and more stable response times under load 3. Significantly lower CPU usage on front-end servers 4. Dramatically higher load carrying capacity of a single instance

Overall, these improvements present compelling reasons for customers to upgrade to Content Server 10.0.

Document Title


Introduction This paper describes the testing efforts undertaken by OpenText to determine the key performance characteristics of Content Server 10.0 relative to those of Content Server 9.7.1. For the most part, the same test method used in previous reports of this type has been used for this release, with the addition of several new tests. The test suite includes isolated operation tests, load tests, and web services ingestion and retrieval tests.

Objectives The objectives of these tests are to:

1. Quantify the difference in performance characteristics between the current release of Content Server (10.0), and the previous release of Content Server (9.7.1),

2. Highlight the performance improvements introduced in the current release of Content Server (version 10.0).

As a result of these tests, we wish to know how the current release of Content Server (10.0) performs for a variety of functions, compared to currently supported releases of Content Server (9.7.1):

• When executed in isolation. • Under anticipated peak load.

Document Title


Web Services Test Objectives The objectives of these tests are to:

1. Compare throughput and performance for ingesting documents through CWS, for CS 10 and CS 9.7.1.

2. Compare throughput and performance for retrieving documents through CWS, for CS 10 and CS 9.7.1.

Document Title


Tests and Results Meeting the objectives requires two types of testing:

1. Single client thread tests to observe the performance of the software for individual requests.

2. Load tests to observe the maximum achievable throughput of the software in the test environment.

CWS Tests The test is run by an open source test client called soapUI, which makes web service requests through the Content Web Services (CWS) API. The time recorded by the client is the time from when the request is submitted to CWS, until the entire response is retrieved. During the test, summary timing logs are collected from Content Server, and client timings are collected from the soapUI client.

The initial data set includes English and multilingual files sourced from Reuters news feed documents and standard load test documents.

During these tests indexing was enabled, and distributed agents for facet calculation were running. However, the throughput numbers presented below reflect only the time taken to complete the web service calls to store the documents in CS. Indexing and facet calculation are an asynchronous task that trails the ingestion, and will be reported on in a future assessment.

1. Ingestion Isolated Operation Tests Ingesting (creating) documents through CWS is achieved using the DocumentManagement createDocument call.

An initial test was done using one client thread in the soapUI test client; running 100 iterations of the createDocument scenario, to measure average create document time under no load. The table below shows that in this no load scenario, the average time to create a document in Content Server 10.0 is about 20 percent slower than in CS 9.7.1.

Table 1: CWS Functional Test - Summary – CreateDocument

CS 9.7.1 Content Server 10.0

% Difference

Count 100 100

Average (ms) 235.12 279.68 -18.9%

Min (ms) 196 243

Max (ms) 362 351

Std Dev 20.54 24.98

Document Title


Although this document creation time is slower in Content Server 10.0, there are new features in Content Server 10.0 such as facets that create additional overhead for this type of operation. Also, it is important to note that Content Server 10.0 is much more scalable than 9.7.1 for this type of operation, as illustrated in the load test section; when configured with 8 threads, Content Server 10.0 has 30 percent higher throughput and can scale to many additional threads, further improving throughput and overall performance.

Analysis of the CS summary timing logs from these tests shows a significantly higher average number of SQL statements and total SQL execution time for Content Server 10.0 than CS 9.7.1. The table below shows average execution time, average SQL execution time, and number of SQL statements for 9.7.1 and 10.0.

Table 2: CWS Functional Test - Execution Times – Create Document


% Difference

Avg Exec Time (sec) 0.210 0.257 -0.047

Avg SQL Exec Time 0.032 0.043 -0.011

Avg SQL Count 129 216 87

2. Ingestion Load Tests Additional tests were executed ingesting batches of 100,000 documents into CS, with different configurations of CS threads, and with soapUI (client) threads set with at least twice as many threads in order to saturate the server and identify the peak achievable throughput. CS 9.7.1 tests were done with 5 and then 8 CS threads (limitations in CS 9.7.1 mean it is not beneficial to run with more than 8 threads), Content Server 10.0 can scale to use more threads, so tests were done with 5, 8, 16, 24, 32, 40, and 48 threads.

The chart below compares throughput in CS 9.7.1 and 10.0; it illustrates that even at the level of 8 CS threads, throughput in Content Server 10.0 is considerably higher (20.8 docs/second compared to 15.88), and as the number of CS threads is increased in Content Server 10.0, its throughput improves to be more than triple that of CS 9.7.1. With CS running with 8 threads, comparing the average createDocument time from the CS summary timing logs shows 9.7.1 average of 470ms, and Content Server 10.0 23.6 percent faster at 359ms.

Document Title

Figure 1: CWS Load Test – Ingestion - Throughput Content Server 10.0 vs. 9.7.1

CPU usage by the LLServer process is somewhat higher for Content Server 10.0 during the above test with 8 CS threads; this reflects the fact that Content Server 10.0 is processing more documents per second and so better utilizing the available processing power. It also remains close to 10 percent for an extended period after the test, which is a result of the new distributed agent processing facet data. The chart below shows a comparison of CS 9.7.1 and 10.0 CPU usage.


Document Title

Figure 2: CWS Load Test – Ingestion - Content Server Host CPU Usage

Memory usage by LLServer starts at a much higher level, and shows a much larger increase during the Content Server 10.0 test, increasing by about 400MB, compared to the minimal 30MB increase during the CS 9.7.1 ingestion. Note that about 1.5 hours after the test completed, Content Server 10.0 memory use decreased by about 100MB. The larger initial memory footprint is partly due to the Content Server 10.0 process being 64-bit (whereas 9.7.1 is 32-bit), and is also due to additional per thread caching in Content Server 10.0. The chart below compares LLServer private bytes memory usage for CS 9.7.1 and 10.0 during the 8 thread ingestion load test.


Document Title

Figure 3: CWS Load Test – Ingestion – Content Server Memory Usage

SQL Server CPU usage is somewhat higher during the Content Server 10.0 test, averaging around 18 percent, but with spikes as high as 50 percent. During the CS 971 test, SQL Server CPU usage was more consistently around 10-18 percent with no spikes higher than 50 percent. Again, this reflects the fact that Content Server 10.0 is processing more documents per second, and that the additional features in Content Server 10.0 mean that for each document, more SQL statements are executed.

During an ingestion batch with 8 CS threads, CPU usage on the 10.0 Admin Server is about twice as high as on the 9.7.1 Admin server. The chart below compares Admin server CPU usage between the two CS versions. CPU usage continues on the Admin server well after the ingestion has completed due to indexing of the ingested documents; as mentioned earlier, further analysis of indexing will be provided in a future assessment.


Document Title

Figure 4: CWS Load Test – Ingestion - Admin Server Host CPU Usage

Memory usage is also higher on the Content Server 10.0 Admin server, with only about 1GB available memory at the start of the test and remaining at that level through the test. For CS 9.7.1, the Admin server starts with about 4.1 GB available; this drops to about 3.9 GB by the end of the test. This is largely due to Content Server 10.0 processes being 64-bit, and partly to additional features in Content Server 10.0.

Additional Content Server ingestion batches were executed with CS threads increased to 16, 24, 32, and 40. CPU usage on the CS server increases with each test until it is about 75 percent for the 40 thread test. The chart below compares CS CPU usage during these tests.


Document Title

Figure 5: CWS Load Test – Ingestion - Content Server Host CPU Usage for Varying Thread Counts

As the CS thread count is increased, the increase in private bytes memory use by LLServer also increases. For tests with more than 8 threads, there is also a large jump in private bytes of 200 MB or more right at the start of the load test; the server is configured for a minimum of 8 threads, so this jump is likely due to initialization of the additional threads at the start of the test. As mentioned earlier, the increased memory usage with additional CS threads is due to additional per thread caching in Content Server 10.0.


Document Title

Figure 6: CWS Load Test – Content Server Memory Usage for Varying Number of Threads

3. Retrieval Isolated Operation Tests Retrieving documents through CWS is achieved using the DocumentManagement getVersion and then getVersionContent call. The soapUI retrieval script loads log files from the previous ingestion tests, and randomly selects ingested document ID’s to retrieve.

The data set includes English and multilingual Reuters files, and also includes 1 million documents loaded by the ingestion test. The Admin server had only search engines running, and no index engines running.

Tests were first run using one client thread in the soapUI test client, running 500 iterations, to obtain average retrieval times without any load on the server. The table below shows that in this no load scenario, Content Server 10.0 is 54 percent faster than 9.7.1 for the getVersion call, and 20 percent faster for getVersionContents.


Document Title


Table 3: CWS Functional Test - Client-Side Timings - Retrieval


% Difference

getVersion 116.88 53.7 54%

getVersionContents 162.36 128.79 20%

Combined 279.24 182.49 35%

4. Retrieval Load Tests The data set includes English and multilingual Reuters files, and also includes 1 million documents loaded by the ingestion test.

Retrieving documents through CWS is achieved using the DocumentManagement getVersion and then getVersionContent call. The soapUI retrieval script loads log files from the previous ingestion tests, and randomly selects ingested document ids to retrieve. Tests were executed for a duration of 15 minutes, with different configurations of CS threads, and with soapUI (client) threads set with at least twice as many threads in order to saturate the server and identify the peak achievable throughput. CS 9.7.1 tests were done with 5 and then 8 threads (limitations in CS 9.7.1 mean it is not beneficial to run with more than 8 threads). Content Server 10.0 can scale to use more threads, so tests were done with 5, 8, 16, 24, 32, 40, and 48 threads.

The chart below compares throughput in CS 9.7.1 and 10.0. It illustrates that even at the level of 5 CS threads, throughput in Content Server 10.0 is considerably higher (40.8 docs/second compared to 22.61), and as the number of CS threads is increased in Content Server 10.0, its throughput improves to be more than five times that of CS 9.7.1.

Document Title

Figure 7: CWS Load Test – Retrieval –Throughput Content Server 10.0 vs. 9.7.1

While Content Server 10.0 achieves much higher retrieval throughput than 9.7.1 with the same number of threads, it does so with lower CPU usage. The chart below shows a comparison of CS 9.7.1 and 10.0 CPU usage (overall CPU use on the server which includes CS and CWS) for the test with 8 CS threads. During this 15 minute test, a total of 64,012 documents were retrieved from Content Server 10.0, while 29,972 were retrieved from 9.7.1.


Document Title

Figure 8: CWS Load Test – Retrieval – Content Server Host CPU Usage

For Content Server 10.0, as the number of CS threads are increased, overall CPU usage increases until it levels off around 70 percent during a test with 40 CS threads. The chart below illustrates overall CPU usage on the CS server for tests from 5 to 40 threads.

Figure 9: CWS Load Test – Retrieval – Content Server CPU Usage for Varying Number of Threads


Document Title

Memory usage by LLServer starts at a much higher level, and shows a larger increase during the Content Server 10.0 test, increasing by about 40 MB, compared to the minimal 4 MB increase during the CS 9.7.1 15 minute retrieval test. As described in the ingestion section, the larger initial Content Server 10.0 memory footprint is due to it being 64-bit, and to additional per thread caching in Content Server 10.0. The chart below compares LLServer private bytes memory usage for CS 9.7.1 and 10.0 during the 8 thread ingestion load test.

Figure 10: CWS Load Test – Retrieval – Content Server Memory Usage

As expected, the increase in Content Server 10.0 private bytes is larger during tests with more CS threads, where a larger number of documents were retrieved. Also, similar to the ingestion tests, for tests with more than 8 CS threads (where min threads was still set to 8), there is a large increase in LLServer private bytes right at the start of the test that is likely due to initialization of the additional threads. The table below illustrates the increase in private bytes for retrieval tests with different numbers of CS threads, and also includes the number of items that were retrieved during the test.


Document Title

Table 4: CWS Load Test – Retrieval – Content Server Memory Usage Private Bytes (MB)

CS Threads Start End Change # Retrieved

5 713 738 25 36783

8 785 825 40 64012

16 955 1031 76 112404

24 1121 1224 103 145063

32 1292 1414 122 166214

40 1455 1589 134 171699

During the retrieval tests, CPU usage on the SQL Server is minimal. The chart below compares SQL Server CPU usage between Content Server 10.0 and 9.7.1 for the 8 thread retrieval test; both are typically below 5 percent. CPU usage during the 10.0 test is 1-2 percent higher than during the 9.7.1 test.

Figure 11: CWS Load Test – Retrieval – SQL Server Host CPU Usage


Document Title


Web Services Test Summary The objectives of the Web Services Tests were to compare the throughput and performance of ingesting and retrieving documents through Content Web Services between Content Server 10.0 and Content Server 9.7.1. These tests were broken down into Isolated Operation Tests, and Load Tests.

While ingestion of a single document through Content Web Services is slower in Content Server 10.0, results of the Content Web Services load tests showed that even at 8 threads, Content Server 10.0 achieves 30 percent higher ingestion throughput than 9.7.1, and a peak throughput of over 53 documents created per second at 40 threads. This is more than three times the peak ingestion throughput of Content Server 9.7.1.

The improvements to Content Web Services retrieval performance in isolation are even more pronounced under load. While at the same number of threads (5), Content Server 10.0 exhibits almost double the retrieval throughput of Content Server 9.7.1, the most significant difference occurs at peak throughput, where Content Server 10.0 achieves over four and a half times the capacity of Content Server 9.7.1 (over 180 docs/second in 10.0, compared to less than 40 docs/second in 9.7.1).

Document Title


Web User Interface Test Objectives The objectives of these tests are to compare throughput and performance for common user requests:

1. In isolation. 2. Under load.

Meeting these objectives requires two types of testing:

1. Isolated Operation performance tests – to measure the performance of requests in isolation.

2. Load tests – to measure the performance of the system under load.

Document Title


Web User Interface Isolated Operation Test The purpose of this test is to isolate the performance of several common requests for the new release of Content Server compared to the current release. These requests include:

1. Browsing 2. Creating Objects 3. Fetching Documents 4. Personal Menu Items 5. Function Menus 6. Searching

The test is run by a proprietary test client that makes HTTP requests to the server. The time recorded by the client is the time from when the request is submitted to the web server, until the entire response has been received. Timing logs from both Content Server and the test client are collected after the test. A typical sequence of steps taken when processing a web request can include:

1. The browser client performs a DNS lookup of the server name and sends the request to the web server.

2. The web server receives the request, starts the gateway (CGI/ISAPI/Servlet), and the gateway forwards the request to Content Server.

3. Content Server executes the request, and returns the response to the gateway. 4. The gateway receives the response from the server, and forwards it to the

browser client. 5. The browser parses the response, executes JavaScript, and renders the final

page.

Document Title

This sequence is depicted visually in Figure 12.

Figure 12: Content Server Request Life Time-Sequence Diagram

The Server Time (depicted in blue) is recorded as the Execute Time in summary timing logs. The Client Time (from the start of the request until the end of the response) is recorded in the logs of the test client. Non-Server Time (depicted in red) is the Client Time minus the Server Time.

1. Browse Requests Accounting for roughly 60 percent of all requests issued to most Content Server instances, browse is by far the most common activity. As browse is usually a part of any task a user wishes to perform, poor browse performance makes many aspects of


Document Title

the system appear slow. Conversely, if browse is fast, many tasks a user performs seem faster.

Figure 13 shows the average Server Time, and Non-Server Time for browse requests of various sizes on Content Server 9.7.1 and 10.0.

Figure 13: Execute & Response Times for Browse Requests

Figure 13 shows significant improvement to the time spent processing browse requests inside the server at several container sizes, while the time spent processing requests outside the server stays relatively constant. This results in an overall improvement delivering the response to browse requests to the client.

2. Create Requests The “Add: object” page is where users fill out the name, description, and other properties needed to create objects of a given type.

Figure 14 shows the average Server Time, and Non-Server Time for create requests of various object types in Content Server 9.7.1 and 10.0.


Document Title

Figure14: Execute & Response Times for Create Requests

These results show that Content Server 10.0 is able to deliver the “Create” pages for objects significantly faster than Content Server 9.7.1. They also show that much of this difference is due to a change in execute time within the server.

3. Fetch Requests Figure 15 shows the average Server Time, and Non-Server Time to fetch documents of various sizes up to 1 GB on Content Server 10.0 and 9.7.1. Note the logarithmic scale on both the Document Size and Time axis (a log-log scale).


Document Title

Figure 14: Execute & Response Times for Fetch Requests

These results show a constant “base time” to execute fetch requests, for documents up to ~1 MB, and a linear increase in time fetching documents greater than ~1MB.


Document Title

4. Personal Menu Requests Figure 16 shows the average Server Time, and Non-Server Time to execute various requests on the “Personal” menu.

Figure 15: Execute & Response Times for Personal Menu Requests

These results show a significant reduction in the amount of time needed to execute these personal menu requests within the server, and a slight increase in the amount of time needed to process these requests in the CGI. The overall effect is a slight decrease in the time needed to deliver responses to these requests to the client.


Document Title

5. Function Menu Requests Figure 17 shows the average Server Time, and Non-Server Time to execute function menu requests.

Figure 16: Execute & Response Times for Function Menu Requests

These results show a dramatic decrease (50ms vs. 325ms, or ~6.5x) in the time needed to execute Function menu requests in Content Server 10.0, as compared to Content Server 9.7.1. The overall effect is that Content Server 10.0 is able to deliver responses to Function menu requests to the client significantly faster (150ms vs. 450ms, ~3x) than Content Server 9.7.1.

6. Search Requests There are a several parameters that may affect the performance of search requests. One is the number of terms in the query, and another is the type of operator used. For this test, we examined six sets of keywords (pulled at random from the keywords associated with documents) presented in three different ways: a simple token query, and right or left-truncation versions of the same queries.


Document Title


Following is the list of terms queried:

1. privata alternativ 2. *vata alternativ 3. privata alterna* 4. aktie andra kvartalet 5. *ie andra kvartalet 6. aktie andra kvarta* 7. för arbetare inom privat sektor ökade 8. *ör arbetare inom privat sektor ökade 9. för arbetare inom privat sektor ök* 10. att kronan kommer att 11. *tt kronan kommer att 12. att kronan kommer at* 13. vid köp från 14. *id köp från 15. vid köp frå* 16. skriver intentia 17. *iver intentia 18. skriver inten*

Queries 1, 4, 7, 10, 13, and 16 are the token queries; queries 2, 5, 8, 11, and 17 are the same sets of tokens, with one of them left-truncated; queries 3, 6, 9, 12, 15, and 18 are the token queries with one of the tokens right-truncated.

Document Title

Figure 18 shows the response times for the same token queries on 9.7.1 and 10.0.

Figure 17: Response Times for Search Queries (All words)

These results show that Content Server 10.0 is able to respond to simple token-based search requests significantly faster than 9.7.1, and that the response time doesn’t seem to be affected by the number of terms in the query, as it did in 9.7.1.


Document Title

Figure 19 shows the response times for the same right-truncation queries on 9.7.1 and 10.0.

Figure 18: Response Times for Search Queries (Right Truncation)

These results show that Content Server 10.0 responds to right-truncation search queries significantly faster than 9.7.1, though both seem to be affected by the number of terms in the query.


Document Title

Figure 20 shows the response times for the same left-truncation queries on 9.7.1 and 10.0.

Figure 19: Response Times for Search Queries (Left Truncation)

These results show that, for the most part, Content Server 10.0 performs as well as or faster than 9.7.1, and that the difference increases as the number of terms increases.


Document Title


Summary The Web User Interface Isolated Operation Tests tested the performance of several user requests through the web interface. These tests were executed in isolation (requests run one-at-a-time, with no other load on the server) in order to determine the effect of changes to these functions on their performance, independent of any other factors.

Overall, these tests showed a slight decrease in performance for download requests, a slight increase in performance for browse and personal menu requests, and significant improvements to performance for create, Functions menu, and search requests.

Document Title


Web User Interface Load Tests The purpose of the load test is to observe how Content Server 10.0 behaves under an extended period of load, as compared to Content Server 9.7.1. A number of changes in Content Server 10.0 raise questions about how they will affect performance.

Some of these changes include:

• Migration to 64-bit address space • Significant reduction in contention within the Content Server engine • A new Distributed Agent infrastructure • Facetted browsing • Multilingual browsing • Use of standard log4cxx interface for logging

These changes present many questions related to performance:

• How do the improvements and new features in Content Server 10.0 affect performance?

• How does Content Server 10.0 scale under load, compared to 9.7.1?

Each of these questions must be addressed separately.

Method The software is loaded until it peaks at less than 50 percent of the available hardware bandwidth (CPU, Disk, Network), for a period of three hours. This simulates an extended typical “peak hour” of usage.

As with previous versions, the load test uses WebLoad1 to simulate several concurrent clients. However, the way WebLoad is used to generate requests has changed from previous tests.

In previous tests, relatively few load generators would be used (three load generators were used for the 9.7.1 test). Each load generator would run a random request, as a random user, as quickly as possible back-to-back. Each load generation client would select a mix of requests that is designed to simulate normal usage. However, this test setup resulted in a maximum of three concurrent requests at any time, and unrealistic user behavior.

1 © 2007 RadView Software Ltd. http://www.radview.com.

Document Title


For the Content Server 10.0 tests, each load generator now “behaves” like a virtual user. The load generator loads an “agenda,” which mimics the way a user might perform a particular operation.

For example, when running the “fetch” agenda, the load generator selects a random document and a random user, then logs on, browses the Enterprise Workspace, browses the folder tree down to the document, clicks the Functions menu for the document, fetches the document, and finally logs off.

Between each of these steps is a delay that represents the browser parsing and rendering the response, and the user deciding what link to click next (the “think-time”). After this set of steps, the load generator sleeps for a while (the “pace-time”), simulating a user working with the document.

Overall, the result is that the tests more accurately simulate the way users would be expected to use Content Server, so as to better simulate how Content Server can be expected to perform under realistic load patterns.

Load Test Results The purpose of this test is to answer the question: “How does the version of the Content Server affect the performance of end-user requests between Content Server 10.0 and 9.7.1?”

To answer this question, we configure a front-end server with instances of Content Server 9.7.1 and 10.0. Each instance is configured with eight worker threads (a common value used by customers in Content Server 9.7.1), and all agent threads are disabled. We also configure back-end instances for search on another machine. The database for each instance is hosted on the same database server.

Document Title

Figure 21 shows the logical configuration of machines used for this test.

Content ServerFront-End

DatabaseServer

SearchServer

Load Monitor& Controller

Load Test Logical Network Diagram

Figure 20: Load Test Logical Network Diagram


Document Title

Load Size Summary Figure 22 shows the load profile used for each load test.

Figure 21: Load Profile, 9.7.1 & 10.0 – 8 Threads

The test starts by ramping up approximately 12 load generators every minute until 440 load generators have been started. Delaying the load generator start-ups helps to “de-synchronize” the virtual users.


Document Title

Throughput One aspect of performance is throughput. Throughput is concerned with the total volume of work; it is a measure of how much work can be done in a given amount of time.

Figure 23: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows the throughput in pages per second, as measured in the load generation software over the duration of the test. A “page” is an explicit request to the Content Server gateway (livelink.exe) by the load test script, and does not include references to the /support/ directory mapping.

Figure 22: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads


Document Title

Table 5 represents a summary of the data as depicted in Figure 23 for the peak user load (excluding the periods of ramp-up/ramp-down).

Table 5: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads

Name Min Max Average Stdev

Load Size 9.7.1 4.165 440.00 387.89 113.23

Load Size 10.0 5.414 440.00 388.11 112.66

Pages per second 9.7.1 13.40 21.9 17.713 1.578

Pages per second 10.0 11.95 19.1 14.998 1.253

Figure 23: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows that the two tests (against 9.7.1 and 10.0) use the same configuration of load generators (up to 440 virtual users during the peak), but that there are fewer requests being delivered to the 10.0 instance than to the 9.7.1 instance.

In Content Server 9.7.1, and earlier, logon requests, and requests to browse a folder with a customview were served by redirecting the user to the desired URL. This redirect results in two requests being generated for a single “user action.” In Content Server 10.0, this behavior has been changed so that the final link is presented to the user, and the redirects on logon and customview browse are no longer required. The result of these changes is a significant reduction on the number of requests made to the server.

Figure 24: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows the throughput, in transactions per second, as measured from Content Server summary timing logs.


Document Title

Figure 23: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads

Figure 24: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows that there is little difference in throughput during the duration of the tests for the 9.7.1 and 10.0 releases.


Document Title

Execute and Response Times Another important aspect of performance is the responsiveness of a system. Rather than looking at the total amount of work that can be performed in a given time (as with throughput), responsiveness is concerned with how quickly individual requests can be served.

Figure 25: Client-Side Response Time, 9.7.1 & 10.0 - 8 Threads shows the average response time for all requests, as measured in the load generation software, over the duration of the tests. The response time includes the time needed to send the request to the web server, execute the CGI, execute the request inside Content Server, and return the response to the client; they do not include the time needed to parse the DOM of the response, execute any client-side JavaScript, or render the final page that gets displayed to the user (see Figure 13: Content Server Request Life Time-Sequence Diagram for details).

Figure 24: Client-Side Response Time, 9.7.1 & 10.0 - 8 Threads


Document Title


Table 6: Client-Side Response Time, 9.7.1 & 10.0 - 8Threads


Load Size 9.7.1 4.165 440.00 387.89 113.23

Load Size 10.0 5.414 440.00 388.11 112.66

Pages per second

9.7.1 0.264 0.885 0.375 0.059

Pages per second

10.0 0.304 0.795 0.429 0.073

These results show that Content Server 10.0 delivers pages to the end-user slightly slower than 9.7.1 at this level of load, on average. However, it should be noted that the difference in mean execute time between the two tests (54ms) is smaller than the variation within the 9.7.1 test alone (where the standard deviation is 59ms). Both versions still deliver the majority of requests in well under a second, and the stress test in the next section, Figure 44, shows that Content Server 10.0 is able to maintain consistently good response times to a much higher level of load than Content Server 9.7.1.


Document Title

Figure 26 shows the server-side execute time of requests over time for both 9.7.1 and 10.0.

Figure 25: Server-Side Execute Time, 9.7.1 & 10.0 - 8 Threads

These results show a significant rise in execute time on 9.7.1 during the ramp-up period, a steady-state with most requests taking 200 milliseconds to execute during the body of the test, and a corresponding drop off in execute time at the end of the test as load decreases.

The 10.0 results show a very slight increase in execute time as load is increased, achieving steady-state at ~150ms per request throughout the duration of the test, except for a couple of spikes nearing the mean execute time for 9.7.1 of 200 milliseconds. The steady-state execute time on Content Server 10.0 is approximately one-third faster than 9.7.1, for the same volume of requests.

Another way of looking at execute times is to show what proportion of requests are completed in less than a given time. Figure 27 presents the execute time data in this manner.


Document Title

Figure 26: Server-Side Execute Time CFD, 9.7.1 & 10.0u3 - 8 Threads

These results show that Content Server 10.0 is able to execute far more requests faster than Content Server 9.7.1. Table 7 shows time under which the given proportion of requests were executed in the 9.7.1 and 10.0 tests.

Table 7: Execute Time Percentiles - 9.7.1 & 10.0u3 Percentile 90% 95% 99%

Execute Time (seconds)

9.7.1 0.6 0.85 1.15 10.0 0.3 0.45 1.05

From this table, we can see Content Server 10.0 tes 95 nt of that execu percerequests almost twice as fast as 9.7.1, at the same level of load.


Document Title

Resource Usage Resource usage is another important aspect of the performance of a system. A shortage of one or more resources can dramatically affect the performance of a system. Comparing the resource usage of two versions of a system under load can help anticipate changes to the resource usage behavior of their system after upgrade. This can help administrators to plan for extra resources that may be needed, and possibly reduce the use of resources that aren’t required.

A typical Content Server deployment makes use of a large variety of resources. However, bottlenecks typically occur only in a handful of areas. The most common resources to become bottlenecks on front-end servers are:

1. Content Server threads 2. CPU usage on the front-end 3. CPU usage on the database server 4. Memory usage on the front-end

Here we compare usage for each of these resources in the 9.7.1 and 10.0 load tests.

Thread Usage

One of the most common areas to become a bottleneck in a front-end deployment is the number of worker threads. Unfortunately, determining the number of threads needed for a given increase in load is not linear in Content Server 9.7.1. As more requests come in, contention between threads within the engine increases, further increasing the number of threads consumed. This effect can be seen clearly in a plot of thread usage vs. load.

Figure 27: Thread usage vs. Load - 9.7.1

Figure 28 shows an example of this behavior on Content Server 9.7.1. The test delivered a varying amount of load to the server, and observed how many threads


Document Title

were consumed servicing the load. Doubling the load from ~2Tx/s to ~4Tx/s resulted in a roughly linear increase in thread usage, from one to two threads. However, increasing the load from ~4Tx/s to ~6Tx/s resulted in an increase in usage of 1.5 threads; and increasing the load from ~6Tx/s to ~8Tx/s resulted in more than doubling the thread usage (from 3.5 to 8).

Figure 29 shows similar thread usage vs. throughput data for a late beta build of Content Server 10.0.

Figure 28: Thread usage vs. Load - 10.0 β

These results show a much more linear increase in thread usage in Content Server 10.0. Given sufficient hardware to run the threads, this allows Content Server 10.0 to scale to a much larger number of threads, without significant degradation in performance. Coupled with a 64-bit address space, this will allow administrators to run a much larger number of threads in a single instance, reducing the number of instances needed to service a given level of load.


Document Title

Figure 30 shows the thread usage over the duration of the load tests.

Figure 29: Server-Side Thread Usage, 9.7.1 & 10.0 - 8 Threads

This thread usage data shows that, for the same level and pattern of load, Content Server 10.0 exhibits significantly lower thread usage than Content Server 9.7.1. With the linear increase in thread usage vs. load presented earlier, this tells us that Content Server 10.0 should able to handle 25 percent more load than 9.7.1 at 8 threads, and the load handling capacity of the server should continue to increase well beyond 8 threads.


Document Title

CPU Usage

Historically, Content Server is a CPU intensive application. So, another important resource in Content Server deployments is CPU usage, both on the front-end servers, and on the database server. Figure 31 shows the CPU usage over time for both 9.7.1 and 10.0 from the eight thread test.

Figure 30: Front-End Server CPU Usage - 9.7.1 & 10.0 - 8 Threads


Table 8: Front-End Server CPU Usage - 9.7.1 & 10.0 - 8 Threads


Load Size 9.7.1 4.165 440.00 387.89 113.23

Load Size 10.0 5.414 440.00 388.11 112.66

Processor:% Processor Time: _Total (9.7.1) 21.659 41.705 30.042 3.352

Processor:% Processor Time: _Total (10.0) 13.11 23.625 17.519 1.761


Document Title

This chart/table shows that Content Server 10.0 uses significantly less CPU on the front-end server, while servicing the same volume of requests.

The average front-end CPU usage during these tests shows that Content Server 10.0 consumes roughly 40 percent less CPU time than 9.7.1 consumes, for the same level of user load.

Another important aspect of resource usage in Content Server is the CPU usage on the database server. With Content Server instances that service user requests, it is relatively easy and inexpensive to add more front-end instances to handle additional user load. However, scaling up a database server to handle the additional load due to an increase in user requests, either by upgrading the hardware in an existing instance, replacing it with a larger system, or scaling out by clustering the database are all expensive propositions.

Figure 32 shows the CPU usage on the database server during the load tests.

Figure 31: DB Server CPU Usage - 9.7.1 & 10.0 - 8 Threads


Document Title

Table 9 represents a summary of the data as depicted in Figure 32 for the peak user load (excluding the periods of ramp-up/ramp-down.

Table 9: DB Server CPU Usage - 9.7.1 & 10.0 - 8 Threads Name Min Max Average Stdev

Load Size 9.7.1 4.165 440.00 387.89 113.23

Load Size 10.0 5.414 440.00 388.11 112.66

Processor:% Processor Time: _Total 9.7.1 (wlibm3bl04) 5.728 12.97 8.616 1.246

Processor:% Processor Time: _Total 10.0 (wlibm3bl04) 9.485 78.154 41.799 14.284

These results show that Content Server 10.0 requires roughly 40 percent more CPU time on the database server, as compared to 9.7.1. However, much of this increase in load on the database server is attributable to background tasks performed by the distributed agent, rather than front-end load. As will be seen in Figure 42 of the stress test, this usage does not increase linearly with the amount of load on the database server.

Memory Usage

A third key aspect of performance is the memory usage of the process. Content Server 9.7.1 only ships as a 32-bit process, limiting its address space to 2 GB or 4 GB. Memory usage was one of the major factors limiting the number of threads a given process could run (that, along with a lack of increase in throughput beyond roughly eight threads).

Content Server 10.0 is a native 64-bit process, so does not have the same address space limit as 9.7.1 (16TB/18EB depending on the host operating system). Figure 33 shows the memory usage of the llserver process on the front-end server during these tests.


Document Title

Figure 32: Front-End Server memory Usage - 9.7.1 & 10.0 - 8 Threads

Table 10 summarizes the data as depicted in Figure 33 for the peak user load (excluding the periods of ramp-up/ramp-down).

Table 10: Front-End Server memory Usage - 9.7.1 & 10.0 - 8 Threads


Load Size 9.7.1 4.165 440.00 387.89 113.23

Load Size 10.0 5.414 440.00 388.11 112.66

Process:Private Bytes:llserver (9.7.1) 397,788,770 414,453,760 408,259,474 3,508,463

Process:Private Bytes:llserver (10.0) 752,543,740 793,937,920 777,932,181 11,073,164

These results show that Content Server 10.0 uses roughly twice as much memory as Content Server 9.7.1, in the same configuration. While this may seem high, recall that this configuration of Content Server 10.0 can handle significantly more load than Content Server 9.7.1 in the same configuration (see Figure 30: Server-Side Thread Usage, 9.7.1 & 10.0 - 8 Threads). Overall, the result is that Content Server 10.0 uses roughly the same amount of memory needed to handle a given volume of load.


Document Title


Load Test Summary

Results of the baseline tests of Content Server 9.7.1 and 10.0 in nearly identical configurations show that Content Server 10.0 is able to deliver responses to a larger volume of requests faster than Content Server 9.7.1. Coupled with the increased scalability of threads, and address space available in a 64-bit process, a single instance of Content Server 10.0 should be able to handle a significantly higher volume of requests, and respond to them quicker, as compared to previous releases of Content Server.

Document Title

Stress Test Results

The purpose of this test is to determine the peak level of load, where the system is no longer able to handle more load without response times starting to suffer, and to isolate the reason for that limit. To accomplish this, the server is put under a gradually increasing load, until performance starts to suffer.

Stress Test – Content Server 9.7.1 Throughput & Response Time

One aspect of performance is throughput. Throughput is concerned with the total volume of work; it is a measure of how much work can be done in a given amount of time.

Figure 34 shows the throughput, in pages per second, and response time, in seconds, as measured in the load generation software, over the duration of the test for Content Server 9.7.1. A “page” is an explicit request to the Content Server gateway (livelink.exe) by the load test script, and does not include references to the /support/ directory mapping.

Figure 33: Throughput & Response Time – 9.7.1

This result shows a fairly constant page time of less than one second for the first two and a half hours of the test, then a steady climb in response time after that point. Similarly, the amount of throughput the server can handle increases linearly up to about 25 pages per second, at which point increasing load doesn’t result in any gain in throughput. So, the peak load capacity of Content Server 9.7.1, in this configuration, with this mix of requests, is roughly 27 transactions per second.

The next step is to determine whether or not this limitation is caused by a resource bottleneck on the hardware used to host the test, or by some limitation in the software.


Document Title

Resource Usage

Resource usage is another important aspect of the performance of a system. A shortage of one or more resources can dramatically affect the performance of a system.

For this test, we compare performance measurements taken at each of the servers under test to thresholds set by the load generator software to detect common resource shortage issues. The thresholds the software monitors are:

• High CPU Activity (total CPU usage > 75 percent) • High Privileged Mode CPU Activity (privileged mode CPU usage > 30

percent and total CPU usage > 50 percent) • Low Level of Available Memory (available memory < 100MB) • High Level of Disk Activity (disk idle time <5 percent)

Any of these conditions will generate highlights on the corresponding chart, yellow for just past the threshold, red for significantly past the threshold.

Figure 35 shows the front-end server resource usage for the Content Server 9.7.1 stress test.

Figure 34: Front End Resource Usage – 9.7.1

These results show a peak CPU usage on the front-end server of ~55 percent, peak privileged mode CPU usage of ~15 percent, minimum disk idle time of ~400 percent, and minimum available memory of ~15GB. These levels did not trigger any threshold violations, indicating that the throughput of this test was not constrained by hardware resources on the front-end server.

Figure 36 shows the back-end server resource usage for the Content Server 9.7.1 stress test.


Document Title

Figure 35: Admin Server Resource Usage – 9.7.1

These results show peak CPU usage of ~30 percent, peak privileged mode CPU usage of ~1 percent, minimum disk idle time of ~300 percent, and minimum available memory of ~10.5GB. These levels did not trigger any threshold violations, indicating that the throughput of this test was not constrained by hardware resources on the back-end server.

Figure 37 shows the database server resource usage for the Content Server 9.7.1 stress test.

Figure 36: Database Server Resource Usage – 9.7.1

These results show peak CPU usage of ~7 percent, peak privileged mode CPU usage of ~2 percent, minimum disk idle time of 63 percent, and minimum available memory of ~11.3GB. These levels did not trigger any threshold violations, indicating that the throughput of this test was not constrained by hardware resources on the database server.


Document Title

Overall, the resource usage levels on all servers involved in the test were well within the limits defined, indicating that the limitation on throughput was not caused by any resource constraints in the environment, but most likely by some limitation of the software.

Response Time Correlation

In order to isolate the most likely cause of the degradation in page time, we present the page time (in black), the llserver virtual memory size (in dark green) with the top five measurements that most closely correlate to the page time:

1. Memory: % Committed Bytes in Use (front-end) – light green 2. Memory: Available MBytes (front-end) – red 3. SQL Server: Transactions, Longest Transaction Running Time – yellow 4. SQL Server: Transactions: Free Space in tempdb – orange 5. SQL Server: Memory Manager: SQL Cache Memory purple

Figure 37: Response Time Correlation – 9.7.1

The results in Figure 38 show the top five measurements that most closely correlate to the page time for the 9.7.1 stress test. Of these, the Percent Committed Bytes in Use from the front-end server matches the page time almost exactly.

From about 30 minutes into the test, until two and a half hours into the test, the virtual memory size and % Committed Bytes in Use stay relatively constant. At two and a half hours into the test, the virtual memory space allocated to the server increases, but the proportion of memory actually in use stays relatively constant.

By three hours into the test, both the proportion of memory in use, and the time to deliver pages starts increasing. By four hours into the test, the virtual memory size is just shy of 2 GB (the upper limit for a 32-bit process on Windows), so the server cannot increase the amount of virtual memory (which would decrease the % of Committed Bytes in Use).


Document Title

Stress Test – Content Server 10.0 Throughput and Response Time

Figure 39 shows the throughput in pages per second, and response time in seconds, as measured in the load generation software, over the duration of the test for Content Server 10.0.

Figure 38: Throughput & Response Time – 10.0u3

These results show that Content Server 10.0 delivers pages consistently in less than one second for the first six hours of the test. At this point, response time starts to become more inconsistent as load is increased.

The next step is to examine resource usage on the servers to see if there are any resource limitations that may be contributing to this behavior.

Resource Usage

As with the 9.7.1 stress test, we compare performance measurements taken at each of the servers under test to thresholds set by the load generator software to detect common resource shortage issues. The thresholds the software monitors are:

• High CPU Activity (total CPU usage > 75 percent) • High Privileged Mode CPU Activity (privileged mode CPU usage > 30

percent and total CPU usage > 50 percent) • Low Level of Available Memory (available memory < 100 MB) • High Level of Disk Activity (disk idle time <5 percent)

Any of these conditions will generate highlights on the corresponding chart, yellow for just past the threshold, red for significantly past the threshold.

Figure 40 shows the resource usage on the front-end server during the stress test on Content Server 10.0, update 3.


Document Title

Figure 39: Front-end Server Resource Usage – 10.0u3

These results start to show a warning around six and a half hours into the test, which persists until the end of the test. The particular threshold that was triggered is the High Privileged Mode CPU Activity (privileged mode CPU usage > 30 percent and total CPU usage > 50 percent). Neither the amount of available memory nor the proportion of time the disk is idle triggered warnings in this test.

This corresponds roughly to the period where page time starts to deteriorate, however, the CPU usage increases very gradually during the test, while page time fluctuates significantly during the test. This indicates that privileged mode CPU usage on the front-end server may be contributing to increased page time, but that it is not likely the primary cause of the fluctuating page times near the end of the test.

Figure 41 shows the resource usage on the back-end server during the stress test on Content Server 10.0.

Figure 40: Admin Server Resource Usage – 10.0u3


Document Title

These results show peak CPU usage of ~30 percent, peak privileged mode CPU usage of ~5 percent, minimum disk idle time of ~350 percent, and minimum available memory of ~8 GB. These levels did not trigger any threshold violations, indicating that the throughput of this test was not constrained by hardware resources on the back-end server.

Figure 42 shows the database server resource usage for the Content Server 10.0 stress test.

Figure 41: Database Server Resource Usage – 10.0u3

The total CPU usage on the database server during the Content Server 10.0 stress test is significantly higher than during the test on 9.7.1. Much of this increased CPU usage is due to keeping facet counts up to date as content is added to the system. While there are several CPU usage warnings (>80 percent) triggered throughout the test, these do not correspond directly to periods of poor response times.

However, the periods where CPU usage exceeds 90 percent correspond very closely to the spikes in page time observed in Figure 39.

Response Time Correlation – Content Server 10.0

Figure 43 shows the page time (in black) and front-end llserver process virtual memory usage (dark blue) with the top five measurements that correlate most closely to the page time:

1. Memory, % Committed Bytes in Use (front-end) – green 2. Memory, % Committed Bytes in Use (sqlserver) – red 3. Memory, Available MBytes (database server) – yellow 4. Memory, Available MBytes (front-end) – light blue 5. Process, % Processor Time (sqlserver) – purple


Document Title

Figure 42: Response Time Correlation – 10.0u3

These results show that the amount of available and committed memory on the front-end and database servers were most closely related to the increased response times in this test, and CPU usage on the database server was also likely to have contributed to this decrease in response time at the maximum load.

Stress Test Summary Figure 44 shows the results of the two stress tests combined. Page times for both tests are presented in green, and throughput is presented in red. Results from the 9.7.1 stress test are presented with solid lines; results from the 10.0 stress test are presented with dashed lines.


Document Title

Figure 43: Throughput & Response Time – 10.0u3 vs. 9.7.1

The results of the 9.7.1 stress test showed subsecond response times until about two and a half hours into the test. However, after this point, response time increases very quickly as load is added to the system. The lack of any hardware bottlenecks indicates that this is a limitation of the software. Many customers address this bottleneck by installing multiple instance of Content Server 9.7.1 on a single server (vertical scaling), but this significantly increases the effort needed to administer a Content Server environment.

The results of the 10.0 stress test showed subsecond response times until roughly six hours into the test, at two and a half times the user load of 9.7.1. While response times do increase at times beyond this point, they do not show the same linear degradation that was present in the 9.7.1 stress test, and there are clear indications of hardware bottle-necks.

The ability to have a single instance of Content Server 10.0 make use of whatever hardware resources are available (virtually eliminating the need for vertical scaling), significantly reduces the amount of effort needed to administer the environment. Having server resources clearly indicate when there is likely to be degradation in response time makes it easier for administrators to identify and address potential performance issues in a Content Server environment before they become an issue for users.


Document Title


Conclusion The objectives of the Web Services Tests were to compare the throughput and performance of ingesting and retrieving documents through Content Web Services between Content Server 10.0 and Content Server 9.7.1. These tests were broken down into Isolated Operation Tests, and Load Tests.

Results of the Web Services Isolated Operation Tests showed a significant decrease in the performance of creating a document through Content Web Services in Content Server 10.0 as compared to Content Server 9.7.1 (~-19 percent), and a significant increase in performance when retrieving a document (~35 percent).

While ingestion of a single document through Content Web Services is slower in Content Server 10.0, results of the Content Web Services load tests showed that even at eight threads, Content Server 10.0 achieves 30 percent higher ingestion throughput than 9.7.1, and a peak throughput of over 54 documents created per second at 40 threads. This is more than three times the peak ingestion throughput of Content Server 9.7.1.

The improvements to Content Web Services retrieval performance in isolation are even more pronounced under load. While at the same number of threads (5), Content Server 10.0 exhibits almost double the retrieval throughput of Content Server 9.7.1, the most significant difference occurs at peak throughput, where Content Server 10.0 achieves over 4 and a half times the capacity of Content Server 9.7.1 (over 180 docs/second in 10.0, compared to less than 40 docs/second in 9.7.1).

Objectives of the Web U/I tests were to observe how Content Server 10.0 behaves under an extended period of load, as compared to Content Server 9.7.1, and at what level of load the two versions start to show a degradation in response time.

Results of the load test showed that Content Server 10.0 is able to deliver responses to a larger volume of requests faster than Content Server 9.7.1. Results of the stress test showed that the software bottlenecks that prevented Content Server 9.7.1 from using available hardware effectively have been removed from Content Server 10.0, and that a single instance of Content Server 10.0 is able to handle two and a half times the load of an instance of Content Server 9.7.1. By virtually eliminating the need to install multiple instances on a single server (vertical scaling), Content Server 10.0 significantly reduces the amount of effort needed to administer larger scale deployments.

For the most part, results of these tests showed either no loss in performance, or significant improvements to performance with Content Server 10.0. The most notable areas of improvement in Content Server 10.0 include:

5. Improved thread scaling 6. Improved and more stable response times under load 7. Significantly lower CPU usage on front-end servers 8. Dramatically higher load carrying capacity of a single instance

Overall, these improvements present compelling reasons for customers to upgrade to Content Server 10.0.

Document Title

Appendix A: Test Environment Architecture Figure 50 shows the architecture used for the Content Web Services tests. The architecture used for the load tests is shown in Figure 21.

Figure 44: Content Web Services Test Architecture

SQL 2008

Oracle 10

WLIBM3BL02

Content Server and Content Web Services

Sun 7410 Storage

External File Store Hitachi SAN

Index Partitions

OTSM18-08

Content Server Admin (Search)


Document Title


Appendix B: Test Environment Hardware The test environment consists of a Content server with EFS stored on a Sun 7410 storage device accessed through the network, a SQL Server 2008 database server with data stored on SAN attached Hitachi AMS 200 SAN, Admin server with index on SAN attached Hitachi AMS 200 storage device, and a load generation client.

Note that for the Web UI load tests, the EFS was moved to an AMS 2100 SAN device, and the SQL Server database files were also moved to an AMS 2100 SAN device. Specifications for the hardware are listed in the table below:

Table 11: Test Environment Hardware

Content Server EFS Storage Admin Server Index Storage

SQL Server Load Client

Make IBM Sun IBM Hitachi IBM IBM

Model System X 7870AC1

7410 Storage

eServer BladeCentre HS21 8853C2U

AMS 200 SAN

System X 7870AC1

System X 7870AC1

Processors

2 x E5530 Quad Core 2.4Ghz

2 x Quad Core AMD Opteron 2.3GHz

2 x Xeon E5345 Quad Core 2.33GHz



Memory 20GB 16GB 10GB 1GB Cache

20GB 20GB

Network Broadcom BCM5709S NetXtreme II GigE

1Gbps Broadcom BCM5708S NetXtreme II GigE

1Gbps SAN

Broadcom BCM5709S NetXtreme II GigE

Broadcom BCM5709S NetXtreme II GigE

Disk 2 x 136 GB SCSI

11 Seagate ST31000 NSSUN1.OT 932GB (750GB assigned)

5 x 132GB (RAID 0)

2 x 136 GB SCSI 4 x 146GB FC RAID 0

2 x 136 GB SCSI 2

Document Title


Appendix C: Non-Default Configuration Non-Default settings for the CWS Ingestion and Retrieval tests, on CS 9.7.1 and 10.0, are described below.

Table 12: Non-Default Test Settings, all tests Non-default Settings, all tests

Database SQL Server database configured with snapshot isolation and read committed snapshot enabled.

Sessions CS cookie expiry disabled (Server Configuration > Configure Security Parameters > in Cookie Authentication Information set to Never expire).

Table 13: CS10.0 update 3 – Web User Interface and Stress Tests Content Server 10.0 update 3 – Web User Interface and Stress Tests

sockserv;javaserver, running on front-end server, all other agents running on back-end server

Index processes were stopped during these tests.

search partition max content disk size changed from default 5 GB to 50 GB (LPAD-23715).

Table 14: CWS Ingestion and Retrieval tests Content Server 10.0 CWS Tests

Agents Agents and daagent enabled only on Admin server.

Distributedagent enabled only on front-end server.

Patches Sept 2010 monthly patch • Patch to resolve slow performance for CWS document upload (LPAD-23528 ) • Patch for LPAD-23689 • Patch for DA infinite loop issue (LPAD-23386) • After ingestion batch 2, patch to fix long recommender agent query with many new items

Search/

Indexing

After ingestion batch 3, search partition max content disk size changed from default 5 GB to 50 GB (LPAD-23715).

Ingestion batches 1-5 run with 4 index partitions, 6-10 run with 6 partitions.

Ingestion tests run with only index processes running, search processes stopped.

Retrieval tests run with only search processes running, index processes stopped.

Document Title


Table 15: CS 9.7.1 Tests CS 9.7.1

Agents Agents enabled only on Admin server.

Patches June 2010 monthly patch

Document Title


Appendix D: Data-Sets This section describes the data sets used during testing.

Document Sizes Folder: A folder containing various sized files outlined below:

File Name Actual File Size

1k.txt 1 kb

10k.txt 10 kb

100k.txt 100 kb

1m.txt 1000 kb

10m.txt 10 mb

100m.txt 98 mb

1g.txt 977 mb

Folder Sizes Folders: Several Folders, containing different number of files, reflected by the folder name and all sized < 100 KB.

10 Document Folder

100 Document Folder

250 Document Folder

1000 Document Folder

10000 Document Folder

10000 Document Folder Split [Containing 10 folders * 1000 Doc]

100000 Document Folder Split [Containing 9 folders * 11111 Doc]

Document Title


News Article Data-Set The News Article data set consists of ~1.28 million news articles, about 800,000 in English, and about another 480,000 in thirteen other languages. Each article has 12-14 attributes applied, so they have a significant amount of meta-data.

Table 16 shows the category configuration, and count of attribute values.

Table 16: News Article Category Definition Attribute Data Type Length Max. Rows Count

Category Name2 1,277,951

Title Text: Field 254 1 1,277,951

Headline Text: Field 254 1 1,277,951

Byline Text: Field 254 1 1,277,951

Dateline Text: Field 254 1 1,277,951

Copyright Text: Field 64 1 1,277,951

Date Created Date: Field N/A 1 1,277,951

Publisher Text: Field 64 1 1,277,951

Date Published Date: Field N/A 1 1,277,951

Source Text: Field 64 1 1,277,951

Creator Location Text: Field 254 1 798,313

Creator Country Text: Field 254 1 798,313

Topic Text: Field 128 25 4,713,807

Industry Text: Field 128 50 1,485,225

Region Text: Field 128 25 3,192,384

Total 23,767,552

he articles are organized by author, with one folder per author. Each author has full

Tpermissions to their folder, while other authors only have read permissions to their folders. Articles with no author attributed in the source data set are grouped into an “Anonymous” folder. There are 798,313 English articles, grouped into 2393 folders, and 479,638 multilingual articles, grouped into 3 folders.

2 Each item that has a category assigned stores a copy of the category name as its first attribute. This attribute is hidden in the interface, but takes up storage in the database.

Document Title


Table 17 shows the distribution of article content sizes.

Table 17: New Article Size Parameters Parameter Size (Bytes)

Average 1,456

Std.Dev. 1,351

Min 8

Max 61,301

Ingested Document Data Set The documents created during the Web Services ingestion test, and used during the Web Services retrieval test, and general load tests consist of about one million Word documents, with randomly generated text. Each document has a category with 14 attributes applied.

Table 18 shows the category configuration, and count of attribute values for the ingested item category.

Table 18: Ingested Document Category Definition Attribute Data Type Length Max. Rows Quantity

Category Name 1,010,189

DocRef Text:Field 254 1 1,010,189

Email Text:Field 254 1 1,010,189

Approver Text:Field 254 1 1,010,189

RequestedBy Text:Field 254 1 1,010,189

Spec Text:Field 50 1 1,010,189

Code Text:Field 50 1 1,010,189

System Text:Field 100 1 1,010,189

DocDesc Text:Field 254 1 1,010,189

Integer1 Integer: Field N/A 1 1,010,189

Integer2 Integer: Field N/A 1 1,010,189

ReleaseDate Date: Field N/A 1 1,010,189

CompleteDate Date: Field N/A 1 1,010,189

UniqueAttr Text:Field 200 1 1,010,189

FullPath Text:Field 200 1 1,010,189

Total 15,152,835

Document Title


The documents are organized with 1000 per folder. Table 19 describes the size distribution for the ingested documents.

Table 19: Ingested Document Size Parameters Parameter Size (Bytes)

Average 208,011

Std.Dev. 50,419

Min 6,144

Max 450,560

Document Title


Visit online.opentext.com for more information about OpenText solutions. OpenText is a publicly traded company on both NASDAQ (OTEX) and the TSX (OTC) Copyright © 2011 by OpenText Corporation. Trademarks or registered trademarks of OpenText Corporation. This list is not exhaustive. All other trademarks or registered trademarks are the property of their respective owners. All rights reserved.

About OpenText OpenText is the world’s largest independent provider of Enterprise Content Management (ECM) software. The Company's solutions manage information for all types of business, compliance and industry requirements in the world's largest companies, government agencies and professional service firms. OpenText supports approximately 46,000 customers and millions of users in 114 countries and 12 languages. For more information about OpenText, visit www.opentext.com.