appdynamics | appdynamics - how 4 top...

How 4 top universities achieved application intelligence

How 4 top universities achieved application intelligence 2

How 4 top universities achieved application intelligence

Introduction Why higher education needs application intelligence.................................................................................3 Chapter 1 How Cornell University taught its financial system to work smoothly in a production environment..........5 Chapter 2 How Liberty University taught Blackboard systems the meaning of performance.....................................7

Chapter 3 How Washington University mastered the availability of 5.5 million patient records...............................10

Chapter 4 How Missouri State University instructed Blackboard on using visibility to find root causes...................12

Conclusion Why higher Ed needs an APM solution that focuses on transactions, rather than on code.....................13


Most, if not all, universities face enterprise challenges that differ from the average organization. For the most part they depend on third-party applications that are specific to their needs, such as Blackboard LMS (Learning Management System) for course-related activities and Kuali Financial Systems (KFS) to handle university-related actions like financial aid and endowment management.

Introduction: Why higher education needs application intelligence

In administering these distinct duties, universities cannot rely on processes that work for most businesses. Most organizations must contend with overloaded application servers, but few of them face a scenario where 20,000 undergraduates are logging into the same Blackboard application at midnight to register for next semester’s classes. If that app server can’t manage the traffic loads, students find themselves bounced off the network with no explanation, only to discover that the class they were about to register for—and need in order to graduate—has been filled.

This Ticketmaster-style nightmare leads to chaos offline, as these same students overrun the classroom at the opening of the semester, begging for an opportunity to squeak into the class and compel professors into acting as emissaries between these students and an IT department that isn’t designed to deal with such exceptions.

Too often university IT departments lack the insight into a troubled application to solve the problem before it causes such damage. Moreover, tools provided the vendor and open-source point tools do not provide full visibility into all aspects of the university’s IT architecture. A Blackboard outage may be the result of a bug in the app itself, or it may be due to a problem with a referred database, defective hardware, or a virtual machine. So many variables exist that it could takes months before the problem is solved, and IT professionals are stuck putting out fires, rather than making improvements that would add value to the university.

To overcome this seemingly insurmountable predicament, universities need a level of network-wide application intelligence that only a modern, dynamic Application Performance Monitoring (APM) solution can provide. In addition to providing deep insight into the entire network architecture, a comprehensive APM solution like AppDynamics must be extremely intuitive, far-ranging in its capabilities, and able to speak the language of end users, not that of developers. Moreover, it should offer the ability to auto-discover an application’s architecture and quickly alter the mapping of the application’s topology when agile release cycles introduce code changes. And it should be up, running, and instrumenting the distributed application within hours or even minutes.

The four large universities featured in this ebook all faced challenges triggered by a lack of insight into their applications. We hope these examples give you a deeper understanding into why AppDynamics is the best APM solution to manage these specialized applications environments.

Chapter 1How Cornell University taught its financial system to work smoothly in a production environment


Cornell also is the only Ivy League school that is at once public and private because of its unique land-grant heritage. Unlike its Ivy brethren, Cornell has admitted women alongside men since its founding in 1865 and remains dedicated to its mission of accessibility, opportunity, outreach, and public service.

In 2012 Cornell’s IT department deployed Kuali Financial Systems (KFS), a community-sourced enterprise financial services application designed to handle the distinct challenges higher education institutions face while advancing best practices for this important niche. The university’s financial services department planned to use KFS to set up billing and accounting, make purchases, and fill purchase orders. Many of Cornell’s employees depend on this system’s uptime and availability to do their jobs. Moreover, Cornell is a Kuali Foundation member and is invested in KFS’ efficacy and long-term success.

But KFS failed consistently to support many of the complex transactions these employees attempted to perform—despite working effectively in development environments. Critical tasks typically required multiple levels of review and approval, and these extra steps frequently brought about service outages. “We had issues with the system crashing or becoming non-responsive a few times a week,” says Bryan Hutchinson, Cornell’s KFS service manager. “It was challenging trying to pinpoint the root cause of those issues.”

Initially Hutchinson’s team used monitoring applications like JVM profiler YourKit and open-source Java and Java EE tool JavaMelody. However, both tools generated too much overhead to be useful in a production environment. More importantly, these point tools couldn’t supply the system-wide insight Hutchinson’s group needed in the production environment to determine the root cause of a performance problem.

Clearly the Cornell KFS team needed an APM solution that would provide the visibility to minimize MTTR (Mean Time To Repair) without increasing overhead in its production environment.

When the KFS team asked other KFS partner schools and Kuali consultants for a recommendation, both parties recommended the AppDynamics platform. These endorsements, combined with AppDynamics’s history monitoring Kuali applications, inspired the Cornell KFS team to test AppDynamics on its KFS installation.

Almost immediately, the KFS team gained the visibility to correlate symptoms of a problem to its root cause, which drastically reduced the amount of response time they needed to within half a day, according to Hutchinson.

For his part, Kevin Kronenbitter, KFS technical lead at Cornell, stressed the difference between fixing a performance problem in a development environment and doing the same in a production one. “Neither of our [previously used] tools provided the visibility and low overhead the AppDynamics platform does in production,” Kronenbitter points out.

In addition to that coveted visibility, AppDynamics also let Kronenbitter view how transactions performed relative to their respective baselines. Given the size variances among transactions performed within KFS, Kronenbitter found this functionality especially useful because he wasn’t required to set those thresholds himself.

Kronenbitter is especially fond of AppDynamics’ “request snapshots” function. A network request snapshot contains summary data about an individual request, as well as metrics including user experience (Normal, Slow, Very Slow, or Stalled) and any business transaction associated with the request if correlation with an instrumented server side application is available. These snapshots have not only helped Kronenbitter troubleshoot performance problems faster, they have enabled him to optimize KFS code to speed up slow transactions.

“We found the biggest benefit from the outlier request snapshots with our payment request transactions,” Kronenbitter notes. “We were able to identify a bottleneck in the transaction that had many items on it, and we were able to refactor the code to eliminate the requests that weren’t really necessary.”

Overall, AppDynamics has transformed Cornell’s KFS installation into something reliable—and fast. “The AppDynamics platform allows us to see what is going on and identify the issues that could be refactored and made faster,” Hutchinson says. “It lets us bridge the gap between anecdotes from users and actual, actionable information.”

Chapter 1: How Cornell University taught its financial system to work smoothly in a production environment

Cornell University is easily the largest member of the Ivy League, with over 13,500 undergraduate students and 6,000 graduate students. Of the 14 colleges and schools that comprise the university, seven undergraduate and four graduate and professional units are located at the main campus in Ithaca, N.Y. Its two medical graduate and professional schools, Weill Cornell, are located on Manhattan’s Upper East Side, with a second branch of the medical school stationed in Doha, Qatar. The University is building a new Engineering School on New York City’s Roosevelt Island with an expected 2017 completion date.

Chapter 2How Liberty University taught Blackboard systems the meaning of performance


Liberty U runs a sprawling, distributed network that utilizes a complex array of off-the-shelf, customized, and homegrown applications. The university operates a Banner ERP system to handle operations like admissions, academic administration, financial aid processing, and faculty support. It uses a version of Blackboard LMS that has been customized and extended with a host of Blackboard “Building Block” extensions, many of which include Liberty U branding.

The intricacies of Liberty U’s network environment make administering it and ensuring fast and reliable performance for its students a challenge at best. For example, every Sunday night throughout the academic year, a majority of students, both campus and online, log onto the Liberty U network to submit their homework before midnight because all classes share this same deadline.

Unfortunately, Liberty U’s system administrators struggled to identify root causes to a plethora of performance snafus because they lacked insight into application environments. Even when root causes were lurking in the least likely places, the administrators were forced to examine the more likely causes before they were ruled out. This deductive approach usually was time-consuming and ineffective.

Not surprisingly, new software releases and updates generated a great deal of anxiety. Scott Howe, system administrator at Liberty University, says two in particular wreaked havoc on the system. The first, a Blackboard Service Pack Update that required a concurrent Java 1.7 upgrade, caused the university’s app servers to crash constantly.

“Blackboard logs dump a ton of stuff into the Tomcat logs—lots of exceptions and lots of just general diagnostic error handling are thrown in there. It was difficult to make heads or tails of it,” Howe remembers with a shudder.

Howe’s team couldn’t make headway because they lacked the needed visibility to zero in the root causes of the problem. “After two weeks of app servers being restarted multiple times a day, and doubling the number of application servers in the pool, we still were not really making any headway,” Howe says.

As this first scenario was playing out, Howe and a Liberty U colleague, a DBA, attended the Blackboard World conference. There they remembered hearing about AppDynamics’ APM solution. The two men downloaded the trial version to see if it could aid them with this intractable problem.

After an easy installation, the trial version quickly revealed odd spikes in the number of erring transactions, all of which appeared to be related to a common language class. Howe and his coworker showed AppDynamics’ diagnostics to a director of performance at Blackboard. After Blackboard analyzed the diagnostics, the Blackboard director informed them of a known bug in all of the Java 1.7 JDKs that was impacting Liberty U’s application servers. “Blackboard got us in contact with Oracle, and they escalated it through their development ranks and pushed out a patch for us,” says Howe.

Howe told Liberty U management they needed AppDynamics to maintain that visibility needed to effectively monitor Blackboard. However, Howe’s pleas went unanswered because the Java performance problem had been solved. It took a second, more serious outage one year later to finally get management’s attention.

That problem commenced after another Blackboard upgrade was implemented right as the university was migrating the database and application servers to build “an entirely new system with a clean slate,” as Howe describes it. After a quick evaluation, Liberty U bought AppDynamics, and Howe’s team reached a surprisingly speedy resolution.

“The interface AppDynamics provides and the ability from [a single pane of glass] to drill down and do real-time troubleshooting and investigation during an issue sold us,” Howe says. “We didn’t feel the need to look at anything else.”

AppDynamics revealed that Liberty U’s servers were showing a massive amount of throughput and general slowness of the JVM. “AppDynamics was able to show us the entire flow of the transaction—the request coming into the application server, the application server talking out, and then the application server talking with the database. We were able to eliminate large portions of our architecture as not being the source of the problem,” Howe says.

Chapter 2: How Liberty University taught Blackboard systems the meaning of performance

With 13,800 on-campus students and another 100,000 online, Lynchburg, Va.-based Liberty University is the largest private, nonprofit university in the United States and the largest Evangelical Christian university in the world. The university offers more than 450 programs ranging from certificate programs to doctorates.


Howe’s team learned that the root cause of the problem was the storage the application servers were accessing. It was unusually slow, with single NFS calls taking from one to eight seconds at times. With over 100,000 students on a Sunday night trying to submit their homework using Blackboard, this problem led to a bottleneck that brought everything to a standstill.

“AppDynamics was wonderful because we were able to see into the flow of data and figure out exactly what wasn’t our problem,” Howe says. “If we hadn’t had this tool, we might have been spending a ton of cycles, and a ton of time, trying to investigate database issues or something else that wasn’t related.”

AppDynamics now provides Liberty U with the needed visibility to find the root cause of a problem. Instead of having to parse through Blackboard’s voluminous logs, Howe’s team now can trace transactions throughout the network—and discover that, say, a Java bug or a hardware failure, rather than Blackboard, is the cause of a given performance outage.

In addition, AppDynamics provides Liberty U with long-term trending numbers to check response times over specific timeframes, from a single day to an entire quarter or year. Indeed, AppDynamics helped IT bridge a communications gap between IT and management. “When you’re a technical person speaking with somebody who’s more administrative, sometimes bridging that gap can be difficult, and having actual numbers to back up your assertions is priceless,” Howe says. “AppDynamics gives you a ton of information presented in a very legible, clear way.”

Chapter 3How Washington University mastered the availability of 5.5 million patient records


The Biomedical Informatics division at Washington University School of Medicine (WUSM) serves the needs of its researchers and investigators. Its applications provide access to millions of patient records from affiliated hospitals, as well as data on tissue samples, clinical studies and patients outside the university’s network. This data needs to be available instantly to meet the needs of intensive cancer research projects.

One of WUSM’s most important applications is clinical data repository application that includes 5.5 million records of patient data from several affiliated hospitals. Researchers at WUSM, with funding from the National Institute of Health and National Cancer Institute, use this application to identify patient records for their research studies. Therefore, its performance and availability are critical. The more time they have to spend collecting data, the less time they have to conduct potentially life-saving analysis.

Bijoy George, biomedical informatics program manager at WUSM, is responsible for the uptime and availability of many applications in the Biomedical Informatics division, including this electronic repository. The development team that creates and supports this application is not allowed access to the sensitive patient data the application manages. As a result, their QA (Quality Assurance) environment was unable to estimate the load this app would carry in production and lacked the ability to anticipate performance problems. George’s team often spent days—even weeks—on outages and other troublesome issues.

“The application would be running fine, but then all of a sudden it would take more than a minute for a page to load,” George explains. “Because the developers don’t have the data in high-volume, they couldn’t reproduce some of the performance issues.”

George knew his team needed more visibility into this application’s performance, and so WUSM conducted a formal search and evaluation. Ultimately, the university chose AppDynamics as its APM solution.

Soon after George’s team deployed AppDynamics, they found their white whale of bottlenecks in a matter of minutes. By analyzing the path of the application’s key business transactions, they could see that all of the time was being spent in a slow database call. George showed this data to the development team, which quickly repaired it. Pages that previously took minutes to load now loaded within seconds.

“When they saw the call stack analysis, the programmers went ‘Bingo!’ and we were able to solve an issue that had plagued us for a long time,” says George. “Without AppDynamics, I doubt whether even now we would have found the issue.”

George also likes how AppDynamics supplies code-level details related to the execution of a single business transaction. George’s team now can quickly triage a performance problem down to the class and method level—and reduce MTTR from days to minutes.

Moreover, AppDynamics solution also provides George and his team full visibility into their informatics applications, as well as method-level detail into performance problems. The combination of alerting and deep-dive diagnostics allowed George’s team to respond quickly to issues.

“Without AppDynamics, it’s hard to see even what the method does, unless you’re a programmer. But with AppDynamics, we can find out which method is taking the most time and drill down into the details. It’s really helpful in finding issues,” George says.

In addition to the many attributes AppDynamics provides, George has a personal favorite AppDynamics feature: its built-in alerts. With alerts in place, George can easily identify potential performance problems before they occur. “I set up exception monitoring, and the tool immediately tells me if there is a problem,” he says. “Out-of-memory exceptions usually happen when a new user tries to do something with the wrong search parameters. With millions of patient records in the database, a string of bad searches like this can essentially cause serious issues.”

With AppDynamics, George and his team now can guarantee that the physicians and researchers working on important projects can access the data they need as fast as possible.

“We no longer waste time in meetings and on phone calls trying to identify application issues,” George concludes. “We’re very thankful to AppDynamics.”

Chapter 3: How Washington University mastered the availability of 5.5 million patient records

Washington University in St. Louis has boasted a U.S. News & World Report Top 10 medical school since the magazine began publishing rankings back in 1987. In 2013 and 2014, U.S. News ranked it as the top medical school for primary care, while at the same time being a Top 10 medical research institution.

Chapter 4How Missouri State University instructed Blackboard on using visibility to find root causes


MSU enterprise systems administrator Mike McManus says he is responsible for “the care and feeding” of Blackboard. The challenges he faced in maintaining Blackboard’s health revolved primarily around his lack of visibility into the application. “Some of the challenges were in reference to a primary set of monitors that we were told to watch,” McManus says. “But when any of them went askew, I had no idea what was going on. When we encountered a performance hit, I’d see the indicators max out, and then within a few minutes I would start to get calls asking if Blackboard was down.”

Because McManus’ team lacked visibility into a problem’s root cause, they couldn’t supply answers on why Blackboard wasn’t working properly, let alone remediate those issues with any rapidity. For several years McManus sought tools that could enhance the information the monitors were supposed to provide, but he couldn’t find anything that was even close to being cost-effective.

After reading an article about AppDynamics’ unique APM solution, McManus downloaded the 30-day trial version, just as Liberty U’s Howe had. Like Howe, McManus was impressed at AppDynamics’ ease of deployment.

Most importantly, AppDynamics gave McManus a level of visibility he had never before experienced, enabling him to remedy a performance problem before it could affect end users. “I can look at the individual nodes in the cluster, see when one of them is starting to stray from the desired benefits, take it out of the cluster, and then start looking at what’s causing the problem,” McManus says.

McManus’ favorite AppDynamics feature is its monitoring capabilities. “I always have [its dashboard] up on my desktop, and it’s always on one of my monitors that’s on the side where people can look at it as they go by my office,” McManus says.

McManus also appreciates the data AppDynamics gathers to document real-time system performance 24/7. The data lets McManus isolate a problem within selected time ranges via data snapshots and powerful filtering. Furthermore, AppDynamics can easily perform a simple export of that actionable data to show proof of what has transpired, something McManus continues to appreciate, just as Cornell’s KFS team and Liberty U’s Blackboard team does.

“While I can watch it during the day, AppDynamics really comes into play at night or early in the morning. If I get a call from a student saying, ‘Blackboard wasn’t working at 2 AM,’ I can quickly figure out whether it was on MSU’s end or whether it was a user or browser error. And I can show proof,” McManus says.

Adds McManus: “I would definitely recommend AppDynamics to anyone who is responsible for administering Blackboard Learning Management System. It’ll definitely make your life much easier.”

Chapter 4: How Missouri State University instructed Blackboard on using visibility to find root causes

Missouri State University (MSU), the second largest university in the state, enrolls more than 20,000 students across its graduate school, school of agriculture, and six undergraduate colleges. Like Liberty University, MSU uses Blackboard LMS to manage most classroom-related activities and course-related materials. Instructors use Blackboard to post syllabi, assignments, and classroom announcements, while students submit homework and teacher assessments on it. Blackboard is the main conduit for all coursework, both in-class and online, making its performance critical to learning at the university.


Conclusion: Why higher Ed needs an APM solution that focuses on transactions, rather than on code

These four examples illustrate the importance of leveraging an APM solution that can visualize and subsequently map out your entire network, anticipate potential bottlenecks before they can frustrate end users, and provide the data necessary to make informed decisions on the best strategies for optimizing applications and the infrastructure they rely on.

In varying degrees, all four universities used four criteria in choosing the right APM solution for their needs:

1. It must be simple to install and use.2. It must offer full visibility into its entire IT architecture, not just individual

applications. 3. It must supply deep diagnostics into production environments without adding too

much overhead.4. It must understand end user transactions, rather than rely on snippets of code.

AppDynamics met all four criteria for each university profiled in this ebook. Liberty U’s Howe installed a trial version and uncovered spikes in errant transactions with minutes. WUSM’s George succeeded in finding the root cause of a performance problem within an app whose data was off-limits to its QA engineers. MSU’s McManus exported actionable data that showed real-time system performance, so that proof of the root cause was readily available to all parties. Finally, Cornell’s KFS team was able to solve performance outages in its production environment by analyzing the business transactions that were causing them.

That fourth measure may well be the most important feature AppDynamics provides its higher-ed customers. By focusing on end-user transactions, such as when a student tries to register for a class on Blackboard, it makes such actions highly visible to developers as well as systems administrators—while also makes the problem intelligible to those on the business end who manage the purse strings. This level of application intelligence not only changes the efficacy of these specialized and complex apps, it also has the potential to make software and infrastructure improvements whose impact can be felt at every level in the quality of life the university provides.

appdynamics | appdynamics - how 4 top...

Documents