DIRECTV Scores Technology Touchdown with Debut of New APM
Went from after-the-fact investigation to proactive optimization and troubleshooting
Provided application and transaction visibility and troubleshooting not available with previous APM solution
Accelerated application updates by minimizing test-cycle requirements
As Senior Director of IT Operations, Applications Services at DIRECTV, Glenn Trattner and the 200 members of his organization play a key role in maintaining business operations and ensuring customer satisfaction. Among other things, Trattner’s team is responsible for more than 100 internal and customer-facing applications that support the company’s sales, customer care, field service and back office operations.
Powering DIRECTV is a formidable server, applications and infrastructure environment. The IT operations team must monitor and manage close to 6,000 physical and virtual servers that run a mix of Windows, Linux and HP-UX. Applications are mostly built using Java and .NET, but PHP and Node.js applications are also part of the mix.
Challenge: Poor adoption of a tool with negative performance impact
DIRECTV began questioning the value of its previous application performance monitoring (APM) solution. “There were two major issues,” recalls Trattner.
“First, we couldn’t validate whether our previous APM solution was having a positive performance impact or effect on our applications.” At least in part, this was due to the operations team using a myriad of tools and utilities to detect and investigate problems—a process Trattner describes as “after-the-fact investigation.”
In addition, DIRECTV wasn’t getting significant adoption of its previous APM tool outside of its core operations team. “We wanted our developers to dive into the tool and analyze the performance of their applications to help troubleshoot issues,” states Trattner. “The developers never really adopted the previous tools. Instead, it would be us sending screenshots or showing them data and asking them to look into a lot of things, which was time consuming for my team. It’s difficult to give an exact count, but we now have several development teams of key applications such as directv.com, our main Direct Sales application, and our middleware using AppDynamics on a regular basis without prompting from operations.”
Trattner’s team began looking for a better solution. Ease of use, was essential for broader acceptance. In addition, DIRECTV insisted on ease of deployment and a lightweight solution with low performance overhead on the monitored applications. Based on these requirements, Trace3, DIRECTV’s IT consultancy partner, recommended that Trattner look into AppDynamics APM. “Trace3 had experience with the product and a good relationship with the team at AppDynamics. They made introductions and we began to explore the AppDynamics solution,” informs Trattner.
“We’re very comfortable with AppDynamics. We were able to deploy the agent to thousands of application instances with no impact or risk to our production environment.”Glenn TrattnerSenior Director of IT Operations @Applications Services
DIRECTV is one of the world's leading providers of digital television entertainment services, delivering a premium video experience through state-of-the-art technology. The company offers an extensive programming lineup and prides itself on delivering industry-leading customer service to more than 39 million customers in the U.S. and Latin America. In fact, according to the 2014 American Customer Satisfaction Index, DIRECTV is number one in customer satisfaction among cable and satellite providers.
Solution: Immediate visibility during a critical time
The selection process was accelerated in Fall 2013 when DIRECTV began experiencing problems during the first few weeks of its pro football Sunday Ticket offering, a product that allows subscribers to login and watch their favorite teams through mobile applications, on the web, or through their in-home set-top boxes.
“Sunday Ticket places a significant load on our website and infrastructure in a very short period of time—especially at the start of the season,” says Trattner. The company’s senior IT management decided to see if AppDynamics could help. “We were impressed with the visibility and insights provided by AppDynamics and decided to move forward with our purchase and deployment,” says Trattner.
“The root cause of an issue is rarely obvious... However, with AppDynamics, it literally took us two seconds to focus in on the issue.”Glenn TrattnerSenior Director of IT Operations @Applications Services
Today, DIRECTV uses the SaaS version of AppDynamics APM to monitor its Java, .NET, PHP, and Node.js applications. Trattner describes the performance overhead from AppDynamics as “negligible,” and he’s most impressed with the way his team is using APM:
“Today, we’re producing alerts and reports out of AppDynamics—we’re doing more proactive work, rather than just using APM in an after-the-fact reactive state. Plus, with our previous solution, if we were deploying a Java agent, the application would need to go through a significant test cycle in the development and test environments prior to migrating into our production environment. We’re very comfortable with AppDynamics. We were able to deploy the agent to thousands of application instances with no impact or risk to our production environment.”
Trattner recalls numerous examples of AppDynamics’ ability to help quickly resolve issues. For example, DIRECTV uses an Active Decisioning Engine application that customer care reps, sales agents and field service reps use to present the most relevant offers to customers and prospects. This application requires frequent updates and releases, making it difficult to pinpoint the cause of a specific performance issue. As Trattner recalls, “The performance delay was impacting our colleagues’ ability to do their jobs. It was also affecting the customer experience as well as company profitability.”
Troubleshooting distributed applications presents unique challenges. “The root cause of an issue is rarely obvious,” Trattner explains. “For example, component A is talking to component B, B is talking to C, C is talking to D. It’s all happening in very high volumes. However, with AppDynamics, it literally took us two seconds to focus in on the issue. AppDynamics highlighted an unexpected communications thread that was occurring tens of thousands of times a minute. Without AppDynamics, there’s no telling how long it would have taken us to detect the irregular flow.”
AppDynamics has also helped improve collaboration with DIRECTV’s development and testing teams. “With a definitive root cause and documentation, the conversation is much easier now, and there’s much less back and forth deciding who’s going to fix the problem. With AppDynamics, we take a screenshot or send them a report and share it with the team that needs to fix it,” says Trattner.
As for a favorite AppDynamics feature, Trattner likes the dashboards and flow map:
“At a very quick glance, you see a chart of an application’s performance and compare it against its normal baseline. It helps you quickly focus your efforts in the right area to accelerate troubleshooting. And with AppDynamics, we can see all of the monitored applications and transaction flow through the service bus. Being able to visualize traffic behavior makes all the difference.”
“I’d recommend AppDynamics to anyone who runs an IT shop,” he adds. “With AppDynamics, the right people showed up and they had the right knowledge. Whether you’re running a small business website or an enterprise like DIRECTV with hundreds of applications, you’re going to benefit from having insight into the application performance and transaction volumes.”
“Today, we are producing alerts and reports out of AppDynamics—we’re doing more proactive work, rather than just using APM in an after-the-fact reactive state.”Glenn TrattnerSenior Director of IT Operations @Applications Services
Sure, the APM product is impressive, but Glenn Trattner, Senior Director of IT Operations, Application Services at DIRECTV, was most impressed with AppDynamics’ people.
“During our pro football Sunday Ticket rollout, we were doing performance testing on our production infrastructure in the middle of the night. AppDynamics’ experts were on the phone helping us, and we hadn’t even bought the product yet. After we purchased, the AppDynamics head of customer success came out and sat down with me to discuss what we like and opportunities for improving the product. Not only do they listen, they actually follow up and get back to you with answers. I can’t say the same thing about many other vendors we work with.”
“I’d recommend AppDynamics to anyone who runs an IT shop...Whether you’re running a small business website or an enterprise like DIRECTV with hundreds of applications, you’re going to benefit from having insight into the application performance and transaction volumes.”
Glenn Trattner, Senior Director of IT Operations, Applications Services, DIRECTV