It takes a large, sprawling, complicated network and a plethora of off-the-shelf, customized, and built-from-scratch applications to run a school of this size. Blackboard is at the heart of the learning management system, tailored with custom “building blocks” and Liberty University branding throughout.
New software releases and system upgrades are an obvious source of anxiety, as much for Liberty University as for any other software-dependent business. When Liberty implemented two major upgrades — one of software alone and a later one of software plus environment — the result in each case was significant performance problems and headaches for the DBAs and tech staff, because the root causes were so elusively buried in the least likely places. The more likely causes get examined first, which use up a lot of time.
The first performance issue occurred when installing a Blackboard service pack about two years ago. The upgrade required a concurrent upgrade to Java 1.7.
“After the upgrade, we started having app servers crashing constantly,” said Scott Howe, System Administrator at Liberty University. “If you’ve ever looked at Blackboard logs, they dump a ton of stuff into the Tomcat logs — lots of exceptions, lots of just general diagnostic error handling is thrown in there. It’s difficult to make heads or tails of it. We were doing our best, but it was really going nowhere. After two weeks of app servers being restarted multiple times a day, and doubling the number of application servers in the pool, we still were not really making any headway.”
The team just didn’t have the visibility they needed to zero in on the problem. Lack of visibility plagued the Liberty team again a year or so later, when they went through an even bigger upgrade, not only installing another Blackboard service pack but also migrating environments. Again, there were performance issues. But this time, they knew they needed the tools and visibility that AppDynamics provides, sooner rather than later.