It didn’t take long to install — while they were at the conference — and soon they started seeing odd spikes in the number of erring transactions, and all of them appeared to have a common class. It looked like there was an issue with a language class.
“We took what we had to one of the directors of performance at Blackboard at the conference and said, ‘here’s what we’re seeing in AppDynamics,’” Howe said. “‘There seems to be a thread here, have you seen this before?’ After they analyzed it, it ended up being that there was a known bug in all of the 1.7 JDKs, and we were just hitting that. So Blackboard worked with us to get in contact with Oracle, and they escalated it through their development ranks and pushed out a patch for us. Problem solved.”
“At that point we told our management that in order for us to effectively do our jobs, they needed to buy AppDynamics, because we didn’t have enough insight to do our jobs effectively without it.”
Unfortunately, it took a second major performance incident, a year later, to get the administration on board with AppDynamics. This time, though, leveraging AppDynamics earlier led to a much speedier resolution.
The problem started after another Blackboard upgrade was implemented while also migrating the database and application servers — “an entirely new system that was built with a clean slate,” is how Howe described it.
“This time, unlike last time, AppDynamics showed us something very different. We weren’t getting errors, we were just seeing a massive amount of throughput, and general slowness of the JVM...The nice thing was, AppDynamics was able to show us the entire flow of the transaction — the request coming into the application server, the application server talking out, and then the application server talking with the database. We were able to eliminate large portions of our architecture as not being the source of the problem.”
It turned out the storage for the application servers was incredibly slow, with single NFS calls taking from one to eight seconds at times. So imagine what that was like on a Sunday night, with a large portion of 110,000 students hitting the application to submit their homework.
“That was our bottleneck,” Howe explained. “AppDynamics was wonderful because we were able to see into the flow of data and figure out exactly what wasn’t our problem. If we hadn’t had that tool, we might have been spending a ton of cycles, and a ton of time, trying to investigate database issues or something else that wasn’t related.”