Infrastructure to Support Infrastructure to Support Extreme Self-ServiceExtreme Self-Service

Session #20653March 15, 2006

Alliance 2006 ConferenceNashville, Tennessee

2

PresenterPresenter

Carol JordanCarol JordanI have been working on PeopleSoft I have been working on PeopleSoft applications since 1997, and have been at applications since 1997, and have been at Stanford since November 2000. Stanford since November 2000. I am the Infrastructure Manager for the I am the Infrastructure Manager for the team supporting Student and HR Systems. team supporting Student and HR Systems.

In addition to the PeopleSoft products, our In addition to the PeopleSoft products, our group also supports Kronos, Resource 25, group also supports Kronos, Resource 25, OnBase Imaging, Resumix and a custom OnBase Imaging, Resumix and a custom Workflow application.Workflow application.

3

OverviewOverview

Before PeopleSoft:- Limited functionality for students- …using a web-enabled mainframe application- …allowing only 35 concurrent student sessions

With PeopleSoft HR/SA and Portal:- Many functions for students- …and for faculty, advisors and staff- …and no portal sessions are turned away

4

Agenda/ContentsAgenda/Contents

In this presentation:In this presentation:• Extreme self-service defined• Our rough start• Where we are now, how we got here• What we learned

5

Overview: Stanford Overview: Stanford UniversityUniversity Stanford University is a private university Stanford University is a private university located 30 miles south of San Francisco and located 30 miles south of San Francisco and just north of Silicon Valley. just north of Silicon Valley.

There are currently 6,700 undergraduate There are currently 6,700 undergraduate and 8,000 graduate students enrolled, with and 8,000 graduate students enrolled, with 1,775 faculty. 1,775 faculty.

Founded in 1891, the university was Founded in 1891, the university was established by Jane and Leland Stanford in established by Jane and Leland Stanford in memory of their son, Leland Stanford, Jr., memory of their son, Leland Stanford, Jr., who died of typhoid fever at the age of 15.who died of typhoid fever at the age of 15.

6

Overview: Application Overview: Application AdministrationAdministration

The Application Administration team is The Application Administration team is part of Administrative Systems, part of Administrative Systems, supporting many of the applications supporting many of the applications used campus-wide. used campus-wide.

There are three AppAdmin teams, There are three AppAdmin teams, supporting PeopleSoft, Oracle supporting PeopleSoft, Oracle Financials, and Reporting.Financials, and Reporting.

The team supporting PeopleSoft is The team supporting PeopleSoft is made up of 5 System Analysts, 3 made up of 5 System Analysts, 3 Oracle DBAs and 3 Windows System Oracle DBAs and 3 Windows System Administrators. Administrators.

Developers and other support staff are Developers and other support staff are part of a separate team within part of a separate team within Administrative Systems.Administrative Systems.

7

Overview: Application Overview: Application AdministrationAdministration

PeopleSoft products currently PeopleSoft products currently installed:installed:

Campus Solutions 8 SP1Campus Solutions 8 SP1 PeopleTools 8.22.05PeopleTools 8.22.05

Enterprise Portal 8.8 SP1Enterprise Portal 8.8 SP1 PeopleTools 8.44.03PeopleTools 8.44.03

Enterprise Learning Management 8.8 SP1Enterprise Learning Management 8.8 SP1 PeopleTools 8.45.12PeopleTools 8.45.12

All on Oracle 9.2.0.6

8

Extreme Self-Service, Extreme Self-Service, defineddefined

Lots of functionality for students, faculty, advisors and staff

and

Lots of users – close to 35,000 logins on our busiest days

9

Extreme Self-Service, Extreme Self-Service, defineddefinedLots of functionality for students

• File or adjust study list and elect grading options• Request an official transcript to be mailed • Print history of courses and grades• Print Enrollment Certification• Declare major and minor• Apply to graduate• Update addresses, emergency contacts• Maintain FERPA elections• View and accept financial aid• View student bill and print statement• View advisors• View degree progress, milestones, program summary• Coming in March…enter or read course evaluations• Coming in April…pay student bill, add money to card plan• Links to other applications: Student Housing, IT Services bill

(Pinnacle), Office of Development

10

Extreme Self-Service, Extreme Self-Service, defineddefinedLots of functionality

• for Faculty− Review class lists− Send e-mail to students in a class− Submit and change grades− Assign a grade proxy − Review previously-submitted grades− Coming in March…read students’ course evaluations

• for Advisors− View lists of current and past advisees− View study lists, grades and unofficial transcripts for current advisees− Place and release holds of current advisees

• for Staff− View on-line pay statement− Make W-4 elections− Enter direct-deposit elections− Register for training, take on-line tests through PeopleSoft ELM− Link to applications: Kronos, on-line W-2

11

Extreme Self-Service, Extreme Self-Service, defineddefined

Axess Usage Summary: Logins - 2005

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

Jan

Feb

Mar

Apr

May Jun

Jul

Aug Sep Oct

Nov

Dec Jan

To

tal

Lo

gin

s

0

150

300

450

600

750

900

1050

Max

Lo

gin

s p

er Q

uar

ter-

ho

ur

Total logins by day Max Logins Per Quarter-hour

Lots of users:

12

Our Rough StartOur Rough Start

• Upgrade to Version 8 went OK

• First couple of months went OK

• …but the system could not support the sustained demand of our busiest days – the first two days of the new academic year

13

Our Rough StartOur Rough Start

Stanford Daily articles with headlines like, “Axess Problems Plague First Week”

What students saw:• log-in attempts that failed• ‘connection refused’ errors between

HR/SA and Portal• pages that didn’t load

14

Our Rough StartOur Rough Start

CIO magazine article: ‘Big Mess on Campus’

“Stanford's IT was still struggling with integrating the enterprise systems when the newly launched PeopleSoft Web portal (called Axess) crashed last fall…Axess couldn't handle the load of all the returning students trying to log in…”

15

Where we are nowWhere we are now

• We are able to handle high load without issues

• Business offices are planning to roll out new functionality using the Axess portal

•…and we’ve managed to stay out of the papers for a while (fingers crossed)

16

Obligatory Hardware Obligatory Hardware DiagramDiagram

17

How We Got HereHow We Got Here

• Upgraded hardware, added hardware• Server hosting custom single sign-on was the

biggest, earliest bottleneck• Original web servers were old and slow• Got the OK to order new hardware after

capacity issues hit

• Lessons: • Go with your gut…if you’re worrying about it,

fix it; if you can’t fix it, have a contingency plan

• Fight the requests to minimize costs by re-using old hardware

18

How We Got HereHow We Got Here

Web-server changes for HR/SA and Portal• Upgraded JRE• Upgraded Jolt and WebLogic to the latest rolling patch• Reduce network disconnect timeout• Used WebLogic console to fine-tune JVM heapsize

(768M) and thread-count (100) on Windows web servers

• Set Windows PIA service to restart automatically if it goes down

• Installed two logical web-servers per physical server to improve memory utilization under Windows

• Separated administrative and self-service users, to set different time-outs, breadcrumbs, and other configuration parameters for different types of users

19

How We Got HereHow We Got Here

Application-server changes for HR/SA and Portal:

• Upgraded JRE• Upgraded Tuxedo to latest rolling patch• Reduced client-cleanup timeout• For Portal and ELM: implemented shared

application-server cache• For Portal and ELM: re-configured Tuxedo

domains to eliminate spawning – always start the maximum number of PSAPPSRVs− 35 per server for each of two Portal application servers− 8 per server for each of two ELM application servers− 15-25 per server for each of two HR/SA application

servers

20

What we learnedWhat we learned

• We can’t predict load with certainty • High load is loosely tied to the beginning of

the term, when fees are due and students need to know where their classes are

• …so we implemented defensive monitoring to send an e-mail when at 400+ logins per quarter-hour

• Sample e-mail text:The Axess login count as of 00:16, 03/13/2006 is 550 in a 15-minute interval.

21

What we learnedWhat we learned

Because we count logins, we can see patterns in user activity:03/12/2006 22:30 5703/12/2006 22:45 6603/12/2006 23:00 9403/12/2006 23:15 8003/12/2006 23:30 10503/12/2006 23:45 24003/13/2006 00:00 550 03/13/2006 00:15 197 03/13/2006 00:30 181 03/13/2006 00:45 128 03/13/2006 01:00 100 03/13/2006 01:15 90 03/13/2006 01:30 76 03/13/2006 01:45 57

Spring enrollment opened at 12:01am on Monday 3/13

22

What we learnedWhat we learned

Monitor at many levels

• At the web-server layer:• We monitor the load-balanced site• …and individual (logical) web servers• …and the HTTP port used by the load-balancer probe

• At the application-server layer:• We monitor for Tuxedo processes• UNIX SysAdmins monitor memory usage, add swap

space as needed• We monitor for CPU utilization (sample e-mail

follows)

23

CPU monitor – sample e-mailCPU monitor – sample e-mailInfo from sar====================================== Time: 00:40:10 usr%: 77 sys%: 10 wio%: 0idle%: 14

Top output======================================load averages: 4.76, 9.29, 15.18 00:40:10106 processes: 101 sleeping, 1 zombie, 4 on cpu

Memory: 8192M real, 1143M free, 6577M swap in use, 18G swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND26073 a2k_prd 4 0 0 203M 187M cpu0 4:40 10.43% PSAPPSRV 9800 a2k_prd 4 58 0 256M 238M sleep 13:49 8.83% PSAPPSRV10991 a2k_prd 4 50 0 245M 227M sleep 11:18 7.61% PSAPPSRV10819 a2k_prd 4 58 0 267M 251M sleep 13:33 6.82% PSAPPSRV

24

What we learnedWhat we learned

Use your PeopleSoft tools!

• PeopleSoft Performance Monitor for Portal and ELM

• WebLogic console for web servers

25

What we learnedWhat we learned

• Keep Tools up-to-date – watch for BEA updates

• Move AppMessaging to a dedicated host – keep back-end processing out of the transactional infrastructure

• Nobody hits ‘logoff’ – configure your web servers to close idle connections quickly

• Performance testing is mandatory for new projects

• Configuration review, top-to-bottom, with PeopleSoft Consulting was helpful

26

Questions?Questions?

27

ContactsContacts

Carol JordanCarol JordanInfrastructure Manager, Student and HR SystemsInfrastructure Manager, Student and HR SystemsAdministrative SystemsAdministrative SystemsStanford UniversityStanford UniversityE-mail: E-mail: cajordan@stanford.educajordan@stanford.edu

Jeff ButlerJeff ButlerLead System Analyst, Student & HR SystemsLead System Analyst, Student & HR SystemsStanford UniversityStanford UniversityE-mail: jbutler@stanford.eduE-mail: jbutler@stanford.edu

Ross ShimabukuroRoss ShimabukuroSystem Analyst, Student & HR SystemsSystem Analyst, Student & HR SystemsStanford UniversityStanford UniversityE-mail: ryshima@stanford.eduE-mail: ryshima@stanford.edu

This presentation and all Alliance 2006 This presentation and all Alliance 2006 presentations are available for presentations are available for

download from the Conference Sitedownload from the Conference Site

Presentations from previous meetings are also availablePresentations from previous meetings are also available