Fox News

Fox News achieves 10x ROI with AppDynamics in first year

Fox News is one of the largest news websites in the United States with up to a billion page views every month. New content is constantly being uploaded to its content management system (CMS) and delivered on the website via Fox's delivery system. In addition, Fox News provides an API that feeds content to other websites for distribution of its content.

Challenge: Temporary solutions couldn't fix ongoing production problems

Fox News had ongoing production issues for nearly two years before they began looking for an application performance management (APM) tool. “The system was a complete disaster,” said Ryan Jairam, System Administrator at Fox News. Sometimes transactions would take up to 5 minutes to return from the database, and that could have a severe impact on the rest of the application. Solving these problems with nothing more than log files was often impossible. So Jairam and his team did what they could to reduce the effects of long-running transactions on the rest of their application. When they first noticed a performance problem, they'd restart the application, which would only solve the problem temporarily. When it came back, they'd file a support ticket with the vendor and ask their developers to take a thread dump for analysis. Meanwhile, they'd use caching to reduce the number of executions of long-running transactions. “We used Squid to create really long cache times, which helped while we were troubleshooting the problem,” Jairam said. However, exhausting the cache created a different set of problems at Fox News. “Since we had to empty the cache once it became stale, we actually created tools that would listen for new RSS feed content, and automatically flush the cache,” he said. “It was terrible, but it worked.” Application problems were amplified with new releases or enhancements. Jairam and a team of system administrators and developers would often dedicate up to 12 hours troubleshooting after a major software release. Upgrades in the CMS and delivery systems sometimes took a week or more of dedicated resources to stabilize the system. Eventually the Fox News team decided they needed a tool that would streamline new deployments and alleviate the burden of incessant firefighting.

AppDynamics managed performance in a high-volume production environment

Jairam had several different tools to manage applications before—some were written in-house, others had too much overhead for the environment. Jairam and his team searched for an APM tool that was fit for a high-volume production environment, and found AppDynamics. The AppDynamics APM solution provided Jairam and his team visibility into performance issues, even with remote web service calls. “We have a whole new level of visibility we didn't have before,” Jairam said. “We can see now when our service providers aren't meeting their SLAs and causing our performance to degrade.” For example, an outage with a search provider once caused publishing to hang. “Our CMS vendor couldn't tell what the issue was,” Jairam said. “But almost immediately after looking at our AppDynamics solution we could see that the feeds to our search service provider were down.” In order to remedy the problem, Jairam and his team temporarily bypassed the search feeds, freeing up more computing resources for the rest of the application.

Nearly $165K per year in productivity saved

Troubleshooting application errors at Fox News improved dramatically after installing the AppDynamics platform. Minor code changes that used to take a team of five employees a day to fix now took one person minutes to resolve. Larger system changes, like an application upgrade, would take only a few hours, versus a full week. With the AppDynamics platform in place, Fox News estimated productivity savings of nearly $165K per year. In addition, the AppDynamics platform helped significantly reduce the volume of support tickets coming to Jairam's office. “We used to have six or seven support tickets a day about our applications,” he said. “Now we only have one or two per week, if any, and 99% of those are user error. We've had weeks where we didn't have any.” AppDynamics drastically changed application performance management at Fox News. “It's been a complete night and day difference here,” Jairam said. “Now we have time to focus on new things and not just fight fires all day.”

We used to have six or seven support tickets a day about our applications. Now we only have one or two per week, if any, and 99% of those are user error. We've had weeks we don't have any.