plone tuneup challenges

1

Plone TuneUp challenges

in search of enterprise intranet solution

Andrew Mleczko (redturtle.net)

European Plone Symposium 2009

Sorrento, Italy

2

PoznanPoznan FerraraFerrara

3

● Co-founder of ACC Poland Ltd., the first Polish

Plone company (past)

● Plone consultant at RedTurtle Technology -

based in Ferrara (present)

● Plone-based and Lotus-based web applications

4

RedTurtle Technology

“Sharing knowledge

about knowledge sharing”

5

So what can you expect...

● Plone is just a CMS?

● Case study – Intranet Provincia di Ferrara

● Loadtests environment

● Plone TuneUp challenges

6

How we can use

Plone?

7

Plone variety

● Community portals

● Document management system (DMS)

● Groupware - Collaboration

● Subsite management system

● Enterprise intranets

● ...

● Combination of all mantioned above

8

Plone as CMS

vs.

Plone as framework

9

Why?

“Plone is NOT a general web development framework.“

/Carlos de la Guardia/

10

Plone can be SLOW*

*when using the wrong way

11

Framework comparision

● Used hardware:

– CPU: AMD OpteronT Processor 146 (2 GHz)

– Memory: 2 GB

● Testing with Siege 2.68

● Read scenario

● Results for 50, 200, 300 Concurent Users (CU)

● More details: www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks

12

Comparision

Django RubyOnRails TurboGears0

0,5

1

1,5

2

2,5

50 CU200 CU300 CU

Response time, secs.

13

Comparision

Django RubyOnRails TurboGears0

5

10

15

20

25

30

35

50 CU200 CU300 CU

Longest transaction, secs.

14

So where is Plone?

15

Far, far away

Django RubyOnRails TurboGears Plone0

2

4

6

8

10

12

14

16

18

50 CU200 CU300 CU


Django RubyOnRails TurboGears Plone0

10

20

30

40

50

60

50 CU200 CU300 CU

Longest transaction, secs.

16

But comparing with Zope2

Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope20

2

4

6

8

10

12

14

16

18

50 CU200 CU300 CU


17

What can we do...

● Do not treat Plone as a framework

● For some use cases:

use ContentMirror + fast framework (django,

pylons, repoze)

● But what about enterprise intranet solutions?

18

Plone for enterprise intranet

● Large number of users (more than 500) and

documents (more than 100 000)

● A lot of simultaneous write requests (>20 CU/s)

● Using of caching limited

● Slowest acceptable resp. time <10 seconds

the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]

19

Intranet Provincia di Ferrara

Case study

20

Prerequisites

● 800 users stored in Domino Server (by PAS)

● 170.000 documents

● Yearly growth: ca 25.000 documents

● Peak traffic: 20 CU/s - write transactions

(based on Domino logs)

21

Objective

● User response time (mixed read/write):

– Slowest (5-6 seconds)

– Recommended (2-4 seconds) 90% of all request

– Fast (<2 seconds)

22

Test environment

● Hardware (current):

– 4 dual-core Intel Xeon 2.8 Ghz

– 32 GB RAM

● Amazon EC2 (future)

23

Test environment

● collective.loadtesting

– funkload

– collective.funkload

● PageTemplate Profiler

● mr.bent

● Munin + redturtle.munin

24

class Readonly(testcase.FLTestCase): def setUp(self): self.server_url = 'http://localhost:8080/plone'

def test_ReadOnly(self): server_url = self.server_url self.get(server_url + "/plone/login_form", description="Get /plone/login_form")

self.post(server_url + "/plone/login_form", params=[ ['came_from', server_url + '/plone'], ['form.submitted', '1'], ['cookies_enabled', ''], ['login_name', ''], ['pwd_empty', '0'], ['__ac_name', 'admin'], ['__ac_password', 'admin'], ['submit', 'Accedi']], description="Post /plone/login_form")

collective.loadtesting

25

mr.bent

>>> from mr.bent.wrapper import mkwrapper>>> from mr.bent.tests.test_bent import callcounter>>> from Products.CMFPlone.CatalogTool import \ CatalogTool

>>> mkwrapper(CatalogTool.searchResults, callcounter, "catalogsearches")

26

Test scenario

● Log in

● View random folder

● Create document (2 requests)

● View newly created document

● Publish it

● View home page

● Logout

27

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

5CU20CU60CU

Default clean Plone


28

Default clean Plone


0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU

% Errors (ConflictErrors)

29

Let's start TuneUp and install

the basics:

Varnish + Pound (with sticky session) +

+ 6 ZEO Clients (18 threads) + ZEO Server

30

● Apache – proxy/rewriting urls

● Varnish – cache resources, add http headers

● Pound – balancing with sticky sessison

● 6 ZEO clients

● ZEO Server

31

+ Varnish + Pound + ZEO


0

5

10

15

20

25

30

35

40

5CU20CU60CU


32

+ Varnish + Pound + ZEO


0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU


33

Comparing to default Plone

34


0

5

10

15

20

25

30

35

40

Default PloneImprovment

Comparision

Response time, secs., 20 CU

35

Where are the conflict errors

comming from?mainly portal_catalog

36

Let's try:mount portal_catalog to different ZODB

and increase zodb cache on it

37

● Install redturtle.catalogmount● Add zope-conf-additional:

<zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>

38

● And add zeo-conf-additional:

<filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs</filestorage>

39

+ redturtle.catalogmount


0

5

10

15

20

25

5CU20CU60CU


40

+ redturtle.catalogmount


0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU


41

Maybe there is a more

stable solution:collective.indexing

experimental.catalogqueryplan

archetypes.schematuning

42


0

2

4

6

8

10

12

14

16

5CU20CU60CU

+ collective.indexing + archetypes.schematuning


43

+ collective.indexing + archetypes.schematuning


0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU


44

So what is the improvment?

45

Improvment


0

5

10

15

20

25

30

35

40

fresh Ploneafter TuneUp

Response time, secs., 20 CU

46

Let's try to scale it...

47

ZEO scalability problem


0

5

10

15

20

25

30

35

40

45

1:104:206:1810:30

Response time, secs. 20 CU

48

ZEO scalability problem


0

10

20

30

40

50

60

1:104:206:1810:30

% Errors (ConflictErrors), 20 CU

49

Plone intranet limits

● ZODB is slow when handling concurrent writes

transactions

● Scalability problem using flat ZEO

● Missing 'out of the box' performance

(comparing to other framworks, CMS)

50

How we can solve the problem?

51

Ideas

● RelStorage for scalability problem

● collective.solr

● others ?

● share knowledge at plone-enterprise mailing list