The Reserve Bank of New Zealand is New Zealand’s central bank. It is responsible for operating monetary policy to maintain stable prices and assists in the functioning of a sound financial system for the country. In addition, it supplies New Zealand’s physical money and oversees and operates the country’s payments systems. Dozens of applications are critical to the bank’s daily functioning, allowing support staff and policy analysts to do their work. Greg Perrott, Software Architect at RBNZ, is responsible for the performance of several of these critical applications.
Challenges: No Time for Development
Perrott works on a team of fifteen developers that’s responsible for the support and maintenance of their existing applications as well as providing enhancements and developing new applications. When an end user calls the helpdesk, a member of Perrott’s team must stop what he’s doing and devote a few hours to fixing the problem. “We’d usually get out a workaround within an hour, but sometimes it would take longer depending on how difficult it was to locate the problem. Fixes usually aren’t deployed for a few days or even a week,” Perrott said.
This meant that Perrott’s team was not as productive or agile as it could have been on its other projects. “Our ability to deliver enhancements has been greatly affected by support,” Perrott said. “It gets quite frustrating at times. There were occasionally periods of a week or more when no development progressed because of ongoing support issues.”
One of the ongoing issues that Perrott’s team dealt with was long-running transactions and stalls relating to their .NET services. Because time series data was calculated on-the-fly in application logic and memory (rather than using stored procedures in the database) some requests took a long time to complete. Perrott’s team found they had to extend the time-out period for the client in order to reduce the number of failed transactions. “It wasn’t ideal because the end user still had to wait, but at least that way it would eventually return data,” Perrott said.
Targeting these longer-running transactions was an ongoing project of Perrott’s, but without adequate visibility into the production application, it was a daunting task. “Ever since the system went live, I’ve been balking at the work around simplifying it,” Perrott said. “Without access to production data we simply couldn’t find which areas we needed to redesign in order to improve the slowest services and scale the application.”
Solution: AppDynamics for .NET
When Perrott began his free 30-Day Trial of AppDynamics, the bank had been experiencing an ongoing issue with a service client disconnecting from the services. “We had very little info to go on about what was causing the problem, and we weren’t able to replicate the issue in any other environment. It was very timely that I had started using AppDynamics,” said Perrott. During the trial period, Perrott put AppDynamics in the production environment and almost immediately found the issue and its root cause. “That pretty well sold us on enlisting the solution full-time.”
Benefits: Getting Proactive About Performance and Infrastructure
“AppDynamics provides us with a lot of information that we otherwise wouldn’t have access to without requesting a support account or getting someone from the infrastructure team and standing over their shoulder, instructing them what to do,” said Perrott. “Now we can solve the problems without help from the support or infrastructure teams, and we simply tell them what to fix.”
The result of this additional visibility is that Perrott and his team spend significantly less time troubleshooting than before. “Before we got AppDynamics we would spend about a third of our time on support, and another 20% of our time on QA,” said Perrott. “This didn’t leave much time for enhancements and other development projects. With AppDynamics, we spend about a quarter of the time that we used to finding and fixing things.” Perrott estimates that AppDynamics has saved RBNZ $38,000 in productivity savings by reducing the time developers spend on support activities.
Perrott also noticed that AppDynamics helps him find issues that were going unreported before. “I started finding things that clearly were impacting end users, but either they hadn’t reported them or had reported them in such a way that we were unable to identify the problem,” he said. “Now we’re able to fix those things, and we’re being more effective in our optimization. We’re targeting our .NET services with the most load and the longest response times.”
Perrott thinks that AppDynamics has made a difference in how all infrastructure, development and support function at RBNZ. “I still feel we can get a lot more value from AppDynamics, but there has definitely been an improvement in our ability to provide support, our ability to upgrade the system, and our ability to be more proactive,” Perrot said. “Now we’re aware of problems and can fix them before they impact the business.”
|Example Use Case||Before AppDynamics||After AppDynamics||Benefits|
|Reduction in average MTTR for production incidents||Average 10 hours per incident||Reduced by 65%||$22,713 in productivity savings|
|Reduction in time spent finding and fixing problems in pre-production||Average 9 hours per problem||Reduced by 75%||$16,193 in productivity savings|
|Total 1st Year Savings:||$38,906|
|Developer productivity||Devs spent 34% of time on support||Devs spend more time working on other projects||Improved ability to execute on enhancements and new projects|
|Collaboration with Infrastructure||No info about how resources are used||Dev has data to support claims for more resources||Improved application performance due to increased infrastructure|