Liberty University
Liberty University Offers A Lesson In Performance Problem-Solving
Key Benefits
Visibility into the far corners of the entire network and application environment
Replaces need to analyze volumes of log data
Quickly identifies where the problem is not and zeroes in on where the problem is
Challenges
It takes a large, sprawling, complicated network and a plethora of off-the-shelf, customized, and built-from-scratch applications to run a school of this size. Blackboard is at the heart of the learning management system, tailored with custom “building blocks” and Liberty University branding throughout.
New software releases and system upgrades are an obvious source of anxiety, as much for Liberty University as for any other software-dependent business. When Liberty implemented two major upgrades — one of software alone and a later one of software plus environment — the result in each case was significant performance problems and headaches for the DBAs and tech staff, because the root causes were so elusively buried in the least likely places. The more likely causes get examined first, which use up a lot of time.
The first performance issue occurred when installing a Blackboard service pack about two years ago. The upgrade required a concurrent upgrade to Java 1.7.
“After the upgrade, we started having app servers crashing constantly,” said Scott Howe, System Administrator at Liberty University. “If you’ve ever looked at Blackboard logs, they dump a ton of stuff into the Tomcat logs — lots of exceptions, lots of just general diagnostic error handling is thrown in there. It’s difficult to make heads or tails of it. We were doing our best, but it was really going nowhere. After two weeks of app servers being restarted multiple times a day, and doubling the number of application servers in the pool, we still were not really making any headway.”
The team just didn’t have the visibility they needed to zero in on the problem. Lack of visibility plagued the Liberty team again a year or so later, when they went through an even bigger upgrade, not only installing another Blackboard service pack but also migrating environments. Again, there were performance issues. But this time, they knew they needed the tools and visibility that AppDynamics provides, sooner rather than later.
“When we saw the interface AppDynamics provides and the ability from that one spot to drill down and do real-time troubleshooting...that’s what sold us. We didn’t feel the need to look at anything else.”
Scott HoweSystem Administrator @Liberty University
About Liberty University
Liberty University is the largest private, nonprofit university in the nation, offering over 450 programs from certificates to doctoral degrees. Every Sunday night during the academic year, some large portion of the school’s 110,000 students log onto the university network to get their homework submitted before midnight — because all of the programs have the same deadlines. This is one of the challenging scenarios Liberty’s DBAs have to manage, in addition to keeping the overall learning management system, Banner ERP system, and everything else available and responding quickly for all their constituents.
Solution
Interestingly, as the first scenario was playing out, Scott and a fellow Liberty DBA were at the Blackboard World conference, when one of them recalled the AppDynamics product; they decided to download the trial version and see if it could help them.
“At that point we told our management that in order for us to effectively do our jobs, they needed to buy AppDynamics.”
Scott HoweSystem Administrator @Liberty University
It didn’t take long to install — while they were at the conference — and soon they started seeing odd spikes in the number of erring transactions, and all of them appeared to have a common class. It looked like there was an issue with a language class.
“We took what we had to one of the directors of performance at Blackboard at the conference and said, ‘here’s what we’re seeing in AppDynamics,’” Howe said. “‘There seems to be a thread here, have you seen this before?’ After they analyzed it, it ended up being that there was a known bug in all of the 1.7 JDKs, and we were just hitting that. So Blackboard worked with us to get in contact with Oracle, and they escalated it through their development ranks and pushed out a patch for us. Problem solved.”
“At that point we told our management that in order for us to effectively do our jobs, they needed to buy AppDynamics, because we didn’t have enough insight to do our jobs effectively without it.”
Unfortunately, it took a second major performance incident, a year later, to get the administration on board with AppDynamics. This time, though, leveraging AppDynamics earlier led to a much speedier resolution.
The problem started after another Blackboard upgrade was implemented while also migrating the database and application servers — “an entirely new system that was built with a clean slate,” is how Howe described it.
“This time, unlike last time, AppDynamics showed us something very different. We weren’t getting errors, we were just seeing a massive amount of throughput, and general slowness of the JVM...The nice thing was, AppDynamics was able to show us the entire flow of the transaction — the request coming into the application server, the application server talking out, and then the application server talking with the database. We were able to eliminate large portions of our architecture as not being the source of the problem.”
It turned out the storage for the application servers was incredibly slow, with single NFS calls taking from one to eight seconds at times. So imagine what that was like on a Sunday night, with a large portion of 110,000 students hitting the application to submit their homework.
“That was our bottleneck,” Howe explained. “AppDynamics was wonderful because we were able to see into the flow of data and figure out exactly what wasn’t our problem. If we hadn’t had that tool, we might have been spending a ton of cycles, and a ton of time, trying to investigate database issues or something else that wasn’t related.”
Benefits
The major benefit AppDynamics has delivered to Liberty University is, in a word, visibility. The team could trace transactions through the network and learn that the problem was not with the Blackboard applications, but in one case with Java, and in another, hardware-related.
“When we saw the interface AppDynamics provides and the ability from that one spot to drill down and do real-time troubleshooting and investigation during an issue, that’s what sold us,” Howe said. “We didn’t feel the need to look at anything else.”
And day-in and day-out, the AppDynamics platform gives Liberty University real application intelligence.
“It’s great to have long-term trending numbers...to see what response times have been not only over the last 24 hours, but the last week, the last couple of months,” Howe said. “When you’re a technical person speaking with somebody who’s more administrative, sometimes bridging that gap can be difficult, and having actual numbers to back up your assertions is priceless.”
“There’s a ton of information presented in a very legible, clear way.”
“AppDynamics was wonderful because we were able to see into the flow of data and figure out exactly what wasn’t our problem.”
Scott Howe, System Administrator, Liberty University
