Pizza Pizza
Pizza Pizza reduces troubleshooting time from days to minutes with AppDynamics
Key Benefits
Reduced MTTR from 20-30 hours to 15-30 minutes
Helped to reduce average talk time in call center by 10 seconds
Prevented production outages costing $100,000-150,000 per week
Challenge: Application performance impacted website and call center
Amar Narain, Director of IT at Pizza Pizza, was responsible for making sure the applications were always performing well, especially during peak hours.
“Before we deployed the AppDynamics platform, our solutions were incident-focused only. Now we can actually find the root cause of a problem and fix it.”
Amar NarainDirector of IT @Pizza Pizza
Unfortunately, a lack of visibility into the application layer made this a difficult task. The development team put new code in production every two weeks to support changing promotions, and this often had a severe impact on application performance. Without visibility into what was causing performance bottlenecks, Narain and the development team could not effectively triage outages and slowdowns.
When an outage occurred, it was up to Narain's team to find the problem in the logs. With about 50 Tomcat instances, this exercise was extremely time-consuming. It often took 20-30 employee hours to find the point of failure, and even then the problem was far from being resolved. “We usually attempted to solve problems just by adding more resources - bringing in more Tomcat instances, or raising the number of connections to the databases,” Narain said.
These performance problems were detrimental both to the customers that used the website and the call center agents who placed orders from the phone; the slower the request, the longer it took to place an order and the longer the phone call. “The average talk time on a call should be about two minutes and 20 seconds. Ours was 2:25-2:35, which may not sound like that much more, but ten seconds in this industry is huge,” Narain said. “Ten seconds can mean up to 20-30 less agents helping customers during our peak.”
Call length was further impacted when developers released new code into production. “Almost every Friday evening we would either see a slowdown - talk times would usually be 2:47-2:50 - or something would crash,” Narain added.
While the crashes usually only affected a couple of agents, sometimes they could be much more damaging. “We had a couple of incidents that were pretty costly,” Narain said. “One outage took down our entire call center for one hour on a Friday night. That outage probably cost us $100,000-150,000. That's what pushed us to really understand what was happening.”
Pizza Pizza
Pizza Pizza is the largest pizza chain in Ontario, Canada, with more than 600 restaurants and over 3,000 employees. Customers can order pizza over the phone, through the website and on mobile devices. A call center with more than 400 agents places orders for all of the restaurants through an internal application. These applications handle hundreds of thousands of transactions every week, and must always be up and running to cater to the needs of Pizza Pizza's hungry customers.
AppDynamics provided visibility to speed transactions
Pizza Pizza decided to evaluate application performance management (APM) solutions.
The first solution Narain looked at was too costly and didn't meet their needs. Then Pizza Pizza discovered the AppDynamics platform, and decided it was the best fit for the company.
Once installed, the AppDynamics solution quickly provided value to Pizza Pizza. Within a matter of days Narain and his team were optimizing the database queries that were eating up much of their resources. “One of our problems was how queries from the call center were interacting with the customer database,” Narain said. “The transactions would read and commit even though they weren't writing any data. This caused unnecessary overhead on the application.” He added, “Once we optimized those SQL statements we saw a major difference in the feedback we got from call center managers they have seen a significant improvement in response time when they're taking orders.”
“Our average talk time in the call center was 2:25-2:35 - about 10 seconds above industry average. Now it's down to 2:15-2:20.”
Amar NarainDirector of IT @Pizza Pizza
With the AppDynamics platform, Narain and his team have been able to fix performance problems much faster. What used to take 20-30 hours, now takes only 15-30 minutes. “Before, our solutions were dependent on user feedback and logs and focused only on the current incident, rather than other business process behaviors. Now we can actually find the root cause of the problem and fix it,” Narain said.
Furthermore, Pizza Pizza's agile release cycles no longer impacted application performance. The developers and DBA used the AppDynamics solution to find problems in the code before it was pushed to production each week, preventing slowdowns and outages.
“When the development team first saw the tool, you could see the excitement and confidence in their faces, knowing that now they have a bird's eye view of the systems,” Narain said. “With the AppDynamics platform we can catch problems before they cause an outage. And that means we're saving money, and keeping our customers happy. That's what matters.”
“With the AppDynamics platform we can catch problems before they cause an outage. And that means we're saving money, and keeping our customers happy. That's what matters.”
Amar Narain, Director of IT, Pizza Pizza
