The organization first engaged AppDynamics in May 2013, deciding to run a proof-of-concept across its main security scanners, using AppDynamics in a performance environment to spot issues.
Nitin Thakur, technical operations manager, Cisco Security Technology Group, said, “From our initial evaluation, it was clear AppDynamics was able to provide the functionality we required to cope with the sheer volume of transactions we handle on a daily basis. We evaluated other products on the market, but for us, AppDynamics provided true end-to-end visibility, and we were impressed with the ease of deployment. Crucially, its future roadmap meant the company was best placed to help us scale for future growth.”
After a successful proof-of-concept, AppDynamics was rolled out globally across the organization’s 23 data centers — deploying a total of 15,000 agents — in just one week, and all reporting into a single AppDynamics-hosted SaaS controller.
Previously, Cisco received 88,000 alerts a year from standard monitoring tools, of which only 3,000 were genuine alerts requiring remedial action. Through implementing AppDynamics, Cisco has been able to dramatically reduce false positives over the past twelve months.
“Prior to AppDynamics, our network operations center team was working overtime looking in the wrong places, at the wrong problems. Even if the NOC team only spent an average of five minutes looking at each erroneous alert, that amounts to over 7,000 man hours per year of effort that could be better spent either working on critical issues, or helping to drive innovation forward. Since introducing the AppDynamics platform, we have been able to identify problems we weren’t even looking for. AppDynamics has enabled us to move towards data-driven troubleshooting rather than ‘gut-feels.’ The solution gives us visibility when we need it and the application intelligence to know when things aren’t functioning optimally,” said Thakur.
Since introducing AppDynamics, Cisco has identified unique slow business transactions affecting services, as well as addressing multiple configuration errors. For example, Thakur explained, “We spotted a configuration issue which meant 17 million requests were being sent back to our central hub unnecessarily. With AppDynamics, it took just five minutes to find and fix this issue. We were not even looking for this problem and only identified it when it was presented to us.” Among other benefits, this has resulted in a four-to-five percent increase in memory utilization across the entire platform.
“AppDynamics provides a common language between operations, development, and test. Introducing the platform has helped enable us to move towards a DevOps model, which in turn had a positive impact on employee collaboration,” said Thakur. “The visibility gleaned from AppDynamics has made employees feel more empowered to reach problem resolution and able to influence business outcomes.
“Because we make application releases so frequently, for us, it was critical to have APM in production. AppDynamics gives us great visibility into what is happening on the estate, helping us to dramatically reduce the number of escalations and continue delivering a seamless service to our customers.”
Cisco sees Application Runbook Automation (RBA) as a critical part of the monitoring solution, and intends to roll this out extensively in the near future. “The goal is have no alerts that humans interact with directly, ensuring issues are automatically remediated and routed to development teams where necessary,” said Thakur.
“My goal is to be able to let Chaos Monkey loose and sit back to watch AppDynamics monitor and automate the required remediation,” added Thakur.
Cisco also plans to further integrate AppDynamics with more Security Cloud Operations products.