Transition to Drupal @ SF State

Campus: San Francisco State University

1600 Holloway Ave.San Francisco, CA 94132

Presented By: Kevin O'BrienDivision of Information Technology

http://www.nowarninglabel.com@nowarninglabel on Twitternowarninglabel on drupal.org

Choosing DrupalBegan with search for WCM for DoITTried many (Plone, OpenCMS, Joomla, etc.)Settled on Drupal for functionality and a11y1 virtual server, no real environmentStarted with 4.7x quickly moved to 5.xSome other sites run on Drupal independently

Ancient HistoryPortals required on CSU campuses

• Project Approach (IBM Websphere)– Collaboration between SFSU Team

and IBM Team

– Approach according to the established IBM Method

• An emerging consensus regarding portal development includes the following major best practices and considerations:

– There should be one AND ONLY ONE horizontal portal on campus;

– Portals should be developed iteratively;

– The portal should support “single sign-on”; that is, with a single user id and password each user can access all the applications and data that she or he is allowed to use;

– CSU campuses should consider integration with both legacy systems and CMS (PeopleSoft);

– Courseware management tools should be integrated with the portal;

• From: "http://www.calstate.edu/tis/iam/documents/presentations/2005-simi/SIMI_2005_SF_PresentationFINAL.ppt" and http://its.calstate.edu/systemwide_it_advisory/ITAC_keydocuments/PortalPaper_02012001.doc

What really happened

• Portal took 2.5 years to implement base features. Required constant maintenance

• Other groups/depts. refused to buy-in, saw IBM WebSphere as a “black box”

• Project Manager/Maintainer quit• Said hey “How about going to Drupal,

they have it running successfully, maybe would take 2.5 months” on his way out the door

IBM Websphere

• Java Environment• JDK/JavaEE/JSR 168/WSRP• Portlet Factory• Monolithic• Broke weekly• Few expertise• No community• Only support is PMRs

The A Team• Me – Architect / Developer /

Designer

• (~60 hours/week)

• Supakit “superkid” Kiatrungrit – Developer

• (~50 hours/week)

• Bora Kou – Systems Admin(~25 hours/week on this proj)

• Jason D'Silva – Student Asst. - Developer (10 hours/week)

• Management position was actually vacant during our 2.5 month sprint. So had a part-time interim project manager.

Scrum style agile development

• What we liked:– Quick

turnaround– Weekly

feedback– Clear deadlines– Daily checkins– Learning a lot

• What we didn't like:

– Scope creep– Loss of

resources– Cramming at

end– Not enough time

for testing/QA– Working a lot

Demo [demo] of SF State Drupal Portal [demo] of DoIT WCM

Architecture• 2 Dedicated RHEL 5 boxes run Apache (Xeon 2 ghz + 4 GB RAM)

• 2 Dedicated RHEL 5 boxes run MySQL (Xeon 2 ghz + 4 GB RAM)

• 1 Dedicated box runs MySQL cluster mgmt node (+ MMM replication)

• Load balancer in front + 2 mysql db connection scripts

Big IP Load Balancer

MySQL Cluster 7.x MySQL Cluster 7.xMySQL

Mgmt Node

MySQL DB connection test

script

MySQL DB connection test

script

Apache+PHP 5.2 Apache+PHP 5.2

Drupal & MySQL Cluster :(• Drupal will not run out of the box on ndbcluster tables• Problem: menu_router table and locale_sources tables• Kris Buytaert presented on the topic• A bug was filed with MySQL, rejected because

ndbcluster doesn’t support row lengths of that size• Issues on d.o. filed to address this in core

http://drupal.org/node/391130 and http://drupal.org/node/703916

• So how can we make this work?

Making Drupal run on MySQL Cluster

• Split the menu_router table into smaller, more manageable tables

• Remove the key reference on locale_sources or change the key data type to be something other than text/blob

• Provide documentation on install for the other issues• Full instructions are available here:

http://www.nowarninglabel.com/home/node/42

Useful Links for MySQL Cluster• MySQL Storage Engines• MySQL Cluster• The NDB Storage Engine

How to set up a load-balanced MySQL Cluster

SF State Custom ModulesUser Interface Modules

benefitspps

distributiongroups

emergencycontactpps

financialaidofferletter

financialaidofferletteradmin

militarystatus

namechangeselfservice

nextmuni

officialtranscriptrequest

sfsucalendar

sfsulibraryhelp

sfsunews

studentschedule

Utility Modulesdeletecookies

encryption

langhide

printer

sfsulogout

shibbolethdata

validator

xmlservice

Getting the Community to Develop

Offered two trainings– 1) How to install Drupal on your local– 2) How to create Drupal modules– 3) (Future) How to create web services

Need hand-holding

How to get a Module (portlet) into Portal

First content should be discussed with the oversight group before beginning development to ensure Portal is the correct place for it

Some departments or groups may have a page in Portal that they are responsible for and control the content for

All changes must go through the development cycle

Development cycle

•Develop code•Run unit tests•Check into SVN

•Additional development•Interface testing•Tag in SVN

•Verify automated tests•User acceptance testing•Content is added directly here•Management approval

•Automated tests run•If any tests fail, there is no push to production

•Automated tests•If any tests fail, system is rolled back to previous build

Prepare

Define requirements including functionality and UI

Remember you are working within the Portal template so many UI elements are predefined

Define interfaces to backend systems

The most common interfaces are REST/ WebServices that should follow w3 standards

LDAP (V3)

Determine where portlet will be located and who should have access

Local development

Build portlet

Run unit tests

Unit tests should include all UI components and all backend interfaces

Check into svn

Request dev build

Backend

Work with the data and application owners of the source systems to build the required backend interfaces

Make sure you follow the defined security model if there is one, or work with us to define one if the system you are connecting to has no model defined

Development (DEV)

Run unit tests

Run integration tests, finalize development

Peer review all code

Tag SVN code for test

Request Test build

Code is pulled from SVN into Dev

Test/ Quality Assurance (TEST)

Define test plan and execute all functional tests

Tests must include all functions, UI components, and backend interfaces

Perform user acceptance testing

Perform load/performance testing

Submit automated test suite for Staging and Prod

This test suite must be comprehensive

Make any content only changes directly in Test

A manager must sign off indicating the following:

He/she is aware of all changes going into production and approves them for the specified date

The automated tests sufficiently cover all functionality

All code has been peer reviewed

The load/performance tests prove that the application meets the minimum requirements

Tag code in SVN for roll out

Code is automatically pulled from TEST at 1am and deployed into staging

Staging

Automated test suite is run

If any tests fail, production roll out is not performed

If all tests pass, promotion to Prod occurs automatically at 6am

Production (PROD)

Automated tests are run

If any tests fail, production is rolled back to previous release

Accessibility 508 compliant All pages were run through manual checks Theme is color contrast compliant

Security IP based Access for back-end services Everything is behind a firewall Everything is behind Shibboleth auth Core PHP Module is removed Role based access to pages Daily log checks (in future, archive logs in

central change mgmt repository) Grendel-Scan OWASP standards

Scalability• Handled ~350 concurrent users without

problem• 4,600 visitors/day • 101,000 visits and 574,926 page views

Shibboleth single-sign on & Google Analy.• Servers are way underutilized, use about

15% on norm• Load balance / Auto failover (actually tested)• No outages yet over 3 mos.

Performance• Initial Build took 7.5+ sec. Load times• Used Xdebug to find drupal_http_request

was problem. Used cron to cache• Down to 3.5 sec load times. • Installed APC via PECL (quite easy)• (Make sure to give it enough memory)• Down to 1.5 sec load times. • Ensure Oracle db connects quit if can't

reach

Where we are Now vs. Where we want to be

Now campus websites are fragmented, outdated

Not all sites use Univ. Template, and some don't use it correctly or update it timely

We have a monthly meeting Drupal User Group

Depts. want to put sites in centrally managed highly available Drupal install for easily keeping sites up to date

1 Drupal theme means a consistently updated look

Need installation profile for point & click

Drupal is not the solution for everything (though I love it)

Drupal is not – our document management system

(though it could be someday) We use Sharepoint

– our project management system (again Sharepoint)

– our help desk ticketing system (remedy) It can be made to do all these things, but

other products are out there

The Importance of Community Drupal.org Bay Area / Berkeley / South Bay DUGs IRC: #drupal-support or #drupal On campus: http://drupal.sfsu.edu/drupal Monthly Drupal meetups California Higher Education Group:

http://groups.drupal.org/california-highered

Questions? Contact: kobrien@sfsu.edu @nowarninglabel on twitter http://www.nowarninglabel.com