The World Bank

Challenges

Despite a substantial investment in more than 50 monitoring tools, there was little insight into what was being monitored or why an application or service was experiencing a problem. As a result, application issues tended to set off a cascade of finger-pointing.

Michael Makar, the senior IT manager in charge of enterprise monitoring and performance testing, remembered what it was like, “Executive management would get contacted by vice presidents or a country office director complaining that they couldn’t use a mobile approval application, or they were locked out of an HR application. Applications and services were down all the time. There was a lack of visibility across the different teams, and there was no management reporting. The outage calls could take days and it could take weeks for a root cause analysis to identify an issue with an application.” Without an effective way of communicating about performance issues or resolving them, a culture of blame took root.

“AppDynamics collects so many more metrics than what we saw with other vendors.”

Michael Makar

Senior IT Manager @The World Bank

Solution

In 2015, the bank made a commitment to consolidate its operations with the goal of achieving a single, unified view into everything everywhere, whether the code was running in a data center or on AWS or Azure.

"It was critical for us to modernize and bring in the right tools to make sure that we were proactively monitoring the right applications and the right services, and providing the bank’s executive management with performance-based monitoring from all across the world,” Makar said.

After reviewing possible application performance monitoring solutions, the bank selected AppDynamics. “The AppDynamics value proposition for us was around handling the complexity of our applications. We make heavy use of Java and .NET, and we needed to build a view of those applications across the different APIs, services, and microservices that our teams use today,” Makar said. Visibility into user sessions and the ability to drill down into back-end transactions for troubleshooting was critical. With stakeholders requesting more and more native mobile applications, support for Xamarin as a development platform and views of the user experience on a mobile device were important as well.

“I can’t remember the last time we had a war room scenario. The outages that occur are solved so quickly, we don’t need to go into big war room scenarios anymore.”

Michael Makar

Senior IT Manager @The World Bank

Everyone on the same page

Makar and his team deployed AppDynamics to 400 production applications. “We realized a lot of value from AppDynamics right away,” Makar said. Once the Development Teams were able to see the same data as the Operations Team, the finger-pointing stopped. They no longer had to bother web administrators for data they now had at their fingertips. They could see locate any problem they were having with an application and solve it. “The teams stopped blaming each other after we deployed AppDynamics,” he said.

A self-service model

A self-service model quickly emerged, with teams adding AppDynamics to new applications as they were delivered. “We changed the culture,” Makar said. Application owners, business owners, and ops team leaders were all given the tools they needed to access the information through AppDynamics. If an application went down, they couldn’t blame the Monitoring Team for not telling them. They were responsible for configuring their own metrics and alerts.

Goodbye to war rooms and alert blizzards

The results? Far fewer outages and no more calls in the middle of the night. “I can’t remember the last time we had a war room scenario,” Makar said. “The outages that occur are solved so quickly, we don’t need to go into big war room scenarios anymore.” On a per application basis, the average number of problem tickets has declined 67% since AppDynamics was implemented.

By configuring AppDynamics to send all alerts to PagerDuty, alerts are put through an on-call process and teams are only alerted when there is an issue they need to respond to. This is a big improvement over the blizzard of indiscriminate email alerts and text messages that used to be part of a DevOps engineer’s daily life.

Using data to solve business problems

Makar finds he couldn’t stop talking about the data that is now available. “AppDynamics collects so many more metrics than what we saw with other vendors,” he said. Not only is the data indispensable for troubleshooting, but it also is valuable for everything from capacity planning to monitoring the performance of the bank’s infrastructure over time. “Our CIO came up with multiple different KPIs that he wants around performance, availability, mean time to resolution, and mean time between service failures,” he said. “We’ll be automating these reports on a quarterly basis.”

Makar is particularly excited about the opportunity to use the data to solve business problems. Project and loan-related information that is collected along with user activity can be mined for insight. “As our investment officers go through projects, we’re seeing project IDs and the stage of workflow that they’re in,” Makar said. “We’re able to put together reports for management now that give them a better sense of how the business is performing in real time and where the issues are cropping up. And I believe we’ve only scratched the surface of what we can do with the data analytics capability today.”

A year and a half into the AppDynamics implementation, Makar marvels at the difference the application monitoring solution has made. “AppDynamics has changed the culture so much, I don’t think we could live without it,” he said.