Q2eBanking Implements Customer-Centric Monitoring with AppDynamics
Reduced troubleshooting times by hours and in some cases, days
Increased confidence in releases
Virtually every application that made up the Q2 platform was critical to a customer’s and Q2’s own success. But troubleshooting the highly distributed environment was challenging without a unified APM solution.
Q2 had been using a patchwork of tools to monitor an IIS and C# backend paired with a Node.js and Docker front end, with a heavy reliance on logs. However, neither the logs nor tools were of much use if the problem turned out to be on the customer’s end or with one of the many third-party vendors that banks and credit unions relied on. The need for end-to-end visibility prompted Q2 to look into AppDynamics.
“With AppDynamics, I get a customer-centric view of end-user behavior, making it easier for us to comply with individual SLAs.”Jacob Ramsey, Q2eBanking
Jacob Ramsey, the AppDynamics administrator, said that during the Proof of Concept engineers were able to accelerate root cause analysis for problems that were facing one of Q2’s biggest customers. “We were able to troubleshoot SQL calls that were more expensive than they should be and were conflicting or locking,” Ramsey said. “AppDynamics really sold itself at that point because we were able to improve those stored procedures.”
As the use of Q2’s platform climbed past 250 million customer interactions per month, the company’s dependence on AppDynamics increased as well. Ramsey said he has now instrumented 380 applications with about 1,500 agents. Among the features he has come to appreciate is AppDynamics’ ease of use. “It doesn’t take me more than 10 minutes to instrument an application these days,” he said. Meanwhile, other engineers have quickly become comfortable using AppDynamics, spurring adoption and increasing the overall value realized by Q2.
Q2 was founded in 2004 with a mission to strengthen communities by bolstering their financial institutions. Q2’s smart banking platform allowed banks and credit unions to provide a seamless, personalized experience to every type of customer across all devices, and its behavioral analytics engine helped protect assets and predict customer needs. Its innovative approach brought in new business and monthly user interactions soared. However, as growth continued, performance monitoring became an increasing challenge as Q2 sought to ensure the performance of its own platform and to give each of its customers insight into how their instance was performing under load.
A customer-centric view of performance
One of the major wins from adopting AppDynamics was a newfound ability to view the performance of Q2’s platform from the point of view of each bank’s customers (in other words, the customers of Q2’s customers). Before adding AppDynamics End User Monitoring (EUM), user-reported issues were often the most difficult ones to diagnose. Operations engineers would try to recreate a problem if they could, but many times they remained in the dark. With EUM, they were able to easily get a user’s perspective.
EUM also made it easier to diagnose third-party issues that were affecting Q2’s platform. “Another area where EUM was really useful for troubleshooting was problems with either third-party analytics or CDN providers,” Ramsey said. “A lot of times our application is functioning at a high level, but banks will have a script that is sending data to outside analytics partners. If the analytics piece goes down, it can cause problems with loading our software because it happens further up the chain.”
Locating problems with a blinking red dot
AppDynamics’ dynamic baselines make it possible to create custom alerts that track the performance for each of Q2’s customers, as well as for the overall platform.
Ramsey said Q2 initially tried monitoring applications by creating health rules based on a certain percent of errors, but that approach ended up being too noisy. To reduce false alerts, Ramsey switched to dynamic baselines—these are automatically created by the AppDynamics controller using machine-learning algorithms to account for changes to baseline performance over time. “By incorporating dynamic baselines we were able to drive down the number of false alerts by more than 83%,” Ramsey reported.
“AppDynamics shows me where all our containers are running and gives me full visibility into their resource consumption.”Jacob Ramsey, Q2eBanking
Another way that Ramsey ensures end users will have the best possible experience is to keep tabs on how the platform is performing for different customers. To this end, he autogenerates individual dashboards so each customer has the option of tracking their own performance.
Use of AppDynamics has reduced troubleshooting by hours—and in some cases days— by pinpointing the location of a problem. “It really helps to have a giant, blinking red dot to tell people where to start looking when you have applications spread over 100 different servers and a variety of different applications,” Ramsey said. “Rather than assemble folks from the networking team, storage team, VMware team, and app team in an IT war room where they would typically end up pointing fingers at each other, AppDynamics helps everyone get on the same page.”
Q2 has also seen daily error rates drop using AppDynamics and experienced an increase in overall code quality. “AppDynamics alerted us to a lot of existing problems that we didn’t know we had,” Ramsey said.
Like many organizations, Q2 is transitioning to DevOps and has already begun to increase release velocity. As teams become more agile, AppDynamics is giving them confidence that new releases will not introduce regressions. The assurance is not only important to internal teams, but also helps build trust with banks and credit unions who need to have confidence that each new version of the platform is highly performant before they turn it over to their customers. With AppDynamics, financial organizations can run their own load tests and observe the results themselves.
Visibility into containers
Q2 has gotten additional value from AppDynamics’ server monitoring, Ramsey said. Containerized architectures offer many benefits, including portability, scalability, and resilience. But the ephemeral nature of containers, as well as their plentiful quantity, makes them difficult to monitor. Tracking the location of deployed containers and the resources they consume would be nearly impossible without AppDynamics, Ramsey said.
“We purchased the server visibility module specifically to solve that problem,” Ramsey said. “We had all these Docker containers that we orchestrated with Nomad. They were floating around and had different names. If an application was having problems, it could be really hard to identify what server a particular container was living on at the time and also how that server was performing.” With server visibility, Ramsey just clicks on a container to monitor CPU and memory consumption and then clicks over to the server to see how it is performing.
Instrumenting a container was as easy as installing the agent on a Docker image. “I would say it’s actually easier to instrument containers than regular servers,” Ramsey said.
Q2 is on its way to fully embracing a multicloud strategy. It’s current environment resides in data centers, and some acquired technologies run in private and public clouds. “We currently have one foot in the cloud and one foot in our data centers,” Ramsey said. “We are doing a lot of work to see what it is going to take to move all our applications to the cloud.” He expects to realize even more value from AppDynamics when the migration gets underway. The dynamic baselines will be useful to make sure that the platform’s performance is unaffected by the move, and afterward, the ability to troubleshoot across the new multicloud environment will be integral to Q2’s future success.
“AppDynamics gives us confidence that we are not introducing regressions, even as we increase release velocity.”
Jacob Ramsey, Q2eBanking