Q2eBanking

Jacob Ramsey, the AppDynamics administrator, said that during the Proof of Concept engineers were able to accelerate root cause analysis for problems that were facing one of Q2’s biggest customers. “We were able to troubleshoot SQL calls that were more expensive than they should be and were conflicting or locking,” Ramsey said. “AppDynamics really sold itself at that point because we were able to improve those stored procedures.”

As the use of Q2’s platform climbed past 250 million customer interactions per month, the company’s dependence on AppDynamics increased as well. Ramsey said he has now instrumented 380 applications with about 1,500 agents. Among the features he has come to appreciate is AppDynamics’ ease of use. “It doesn’t take me more than 10 minutes to instrument an application these days,” he said. Meanwhile, other engineers have quickly become comfortable using AppDynamics, spurring adoption and increasing the overall value realized by Q2.

One of the major wins from adopting AppDynamics was a newfound ability to view the performance of Q2’s platform from the point of view of each bank’s customers (in other words, the customers of Q2’s customers). Before adding AppDynamics End User Monitoring (EUM), user-reported issues were often the most difficult ones to diagnose. Operations engineers would try to recreate a problem if they could, but many times they remained in the dark. With EUM, they were able to easily get a user’s perspective.

EUM also made it easier to diagnose third-party issues that were affecting Q2’s platform. “Another area where EUM was really useful for troubleshooting was problems with either third-party analytics or CDN providers,” Ramsey said. “A lot of times our application is functioning at a high level, but banks will have a script that is sending data to outside analytics partners. If the analytics piece goes down, it can cause problems with loading our software because it happens further up the chain.”

Another way that Ramsey ensures end users will have the best possible experience is to keep tabs on how the platform is performing for different customers. To this end, he autogenerates individual dashboards so each customer has the option of tracking their own performance.

Use of AppDynamics has reduced troubleshooting by hours—and in some cases days— by pinpointing the location of a problem. “It really helps to have a giant, blinking red dot to tell people where to start looking when you have applications spread over 100 different servers and a variety of different applications,” Ramsey said. “Rather than assemble folks from the networking team, storage team, VMware team, and app team in an IT war room where they would typically end up pointing fingers at each other, AppDynamics helps everyone get on the same page.”

Q2 has also seen daily error rates drop using AppDynamics and experienced an increase in overall code quality. “AppDynamics alerted us to a lot of existing problems that we didn’t know we had,” Ramsey said.

Like many organizations, Q2 is transitioning to DevOps and has already begun to increase release velocity. As teams become more agile, AppDynamics is giving them confidence that new releases will not introduce regressions. The assurance is not only important to internal teams, but also helps build trust with banks and credit unions who need to have confidence that each new version of the platform is highly performant before they turn it over to their customers. With AppDynamics, financial organizations can run their own load tests and observe the results themselves.

Visibility into containers

Q2 has gotten additional value from AppDynamics’ server monitoring, Ramsey said. Containerized architectures offer many benefits, including portability, scalability, and resilience. But the ephemeral nature of containers, as well as their plentiful quantity, makes them difficult to monitor. Tracking the location of deployed containers and the resources they consume would be nearly impossible without AppDynamics, Ramsey said.

“We purchased the server visibility module specifically to solve that problem,” Ramsey said. “We had all these Docker containers that we orchestrated with Nomad. They were floating around and had different names. If an application was having problems, it could be really hard to identify what server a particular container was living on at the time and also how that server was performing.” With server visibility, Ramsey just clicks on a container to monitor CPU and memory consumption and then clicks over to the server to see how it is performing.

Instrumenting a container was as easy as installing the agent on a Docker image. “I would say it’s actually easier to instrument containers than regular servers,” Ramsey said.

Multicloud monitoring

Q2 is on its way to fully embracing a multicloud strategy. It’s current environment resides in data centers, and some acquired technologies run in private and public clouds. “We currently have one foot in the cloud and one foot in our data centers,” Ramsey said. “We are doing a lot of work to see what it is going to take to move all our applications to the cloud.” He expects to realize even more value from AppDynamics when the migration gets underway. The dynamic baselines will be useful to make sure that the platform’s performance is unaffected by the move, and afterward, the ability to troubleshoot across the new multicloud environment will be integral to Q2’s future success.