DIRECTV Scores Technology Touchdown with Debut of New APM
As Senior Director of IT Operations, Applications Services at DIRECTV, Glenn Trattner and the 200 members of his organization play a key role in maintaining business operations and ensuring customer satisfaction. Among other things, Trattner’s team is responsible for more than 100 internal and customer-facing applications that support the company’s sales, customer care, field service and back office operations.
Powering DIRECTV is a formidable server, applications and infrastructure environment. The IT operations team must monitor and manage close to 6,000 physical and virtual servers that run a mix of Windows, Linux and HP-UX. Applications are mostly built using Java and .NET, but PHP and Node.js applications are also part of the mix.
DIRECTV began questioning the value of its previous application performance monitoring (APM) solution. “There were two major issues,” recalls Trattner. “First, we couldn’t validate whether our previous APM solution was having a positive performance impact or effect on our applications.” At least in part, this was due to the operations team using a myriad of tools and utilities to detect and investigate problems—a process Trattner describes as “after-the-fact investigation.”
In addition, DIRECTV wasn’t getting significant adoption of its previous APM tool outside of its core operations team. “We wanted our developers to dive into the tool and analyze the performance of their applications to help troubleshoot issues,” states Trattner. “The developers never really adopted the previous tools. Instead, it would be us sending screenshots or showing them data and asking them to look into a lot of things, which was time consuming for my team. It’s difficult to give an exact count, but we now have several development teams of key applications such as directv.com, our main Direct Sales application, and our middleware using AppDynamics on a regular basis without prompting from operations.”
Trattner’s team began looking for a better solution. Ease of use, was essential for broader acceptance. In addition, DIRECTV insisted on ease of deployment and a lightweight solution with low performance overhead on the monitored applications. Based on these requirements, Trace3, DIRECTV’s IT consultancy partner, recommended that Trattner look into AppDynamics APM. “Trace3 had experience with the product and a good relationship with the team at AppDynamics. They made introductions and we began to explore the AppDynamics solution,” informs Trattner.
Today, DIRECTV uses the SaaS version of AppDynamics APM to monitor its Java, .NET, PHP, and Node.js applications. Trattner describes the performance overhead from AppDynamics as “negligible,” and he’s most impressed with the way his team is using APM:
“Today, we’re producing alerts and reports out of AppDynamics—we’re doing more proactive work, rather than just using APM in an after-the-fact reactive state. Plus, with our previous solution, if we were deploying a Java agent, the application would need to go through a significant test cycle in the development and test environments prior to migrating into our production environment. We’re very comfortable with AppDynamics. We were able to deploy the agent to thousands of application instances with no impact or risk to our production environment. ”
Trattner recalls numerous examples of AppDynamics’ ability to help quickly resolve issues. For example, DIRECTV uses an Active Decisioning Engine application that customer care reps, sales agents and field service reps use to present the most relevant offers to customers and prospects. This application requires frequent updates and releases, making it difficult to pinpoint the cause of a specific performance issue. As Trattner recalls, “The performance delay was impacting our colleagues’ ability to do their jobs. It was also affecting the customer experience as well as company profitability.”
Troubleshooting distributed applications presents unique challenges. “The root cause of an issue is rarely obvious,” Trattner explains. “For example, component A is talking to component B, B is talking to C, C is talking to D. It’s all happening in very high volumes. However, with AppDynamics, it literally took us two seconds to focus in on the issue. AppDynamics highlighted an unexpected communications thread that was occurring tens of thousands of times a minute. Without AppDynamics, there’s no telling how long it would have taken us to detect the irregular flow.”
AppDynamics has also helped improve collaboration with DIRECTV’s development and testing teams. “With a definitive root cause and documentation, the conversation is much easier now, and there’s much less back and forth deciding who’s going to fix the problem. With AppDynamics, we take a screenshot or send them a report and share it with the team that needs to fix it,” says Trattner.
As for a favorite AppDynamics feature, Trattner likes the dashboards and flow map:
Sure, the APM product is impressive, but Glenn Trattner, Senior Director of IT Operations, Application Services at DIRECTV, was most impressed with AppDynamics’ people.
“During our pro football Sunday Ticket rollout, we were doing performance testing on our production infrastructure in the middle of the night. AppDynamics’ experts were on the phone helping us, and we hadn’t even bought the product yet. After we purchased, the AppDynamics head of customer success came out and sat down with me to discuss what we like and opportunities for improving the product. Not only do they listen, they actually follow up and get back to you with answers. I can’t say the same thing about many other vendors we work with.”