Founded in 1989, Garmin is the leading worldwide provider of navigation devices and technologies. In addition to its popular consumer GPS products, Garmin
also offers devices for planes, boats, outdoor activities, fitness, and cars. The vertical integration of these products keeps the design, manufacturing, marketing and warehouse processes all in house, allowing Garmin to deliver a seamless experience to its users.
Because the end user experience is so important to Garmin and its customers, ensuring the speed of its web and mobile applications is one of its highest priorities. Doug Strick, Internet Application Admin III, is responsible for the performance of several of Garmin’s web applications, including the online store and the applications that power the call center.
Summary: Getting Real-time and Historical Visibility in Production
Garmin used several tools to monitor the performance of their web application and troubleshoot issues when they arose. Most of these tools were open source, like VisualVM, but Garmin found that these tools weren’t powerful enough to deliver the visibility into application performance. “We never had anything that would give us a good deep dive into what was going on in the transactions inside our application – we had no historical data, and we had no insight into database calls, threads, etc.,” said Strick.
“We knew we needed a tool that could constantly monitor our production environment, allowing us to collect historical data and trend performance over time,” said Strick. “Also, we needed something that would give us a better view of what was going on inside the application at the code level.”
Garmin found that AppDynamics, an application performance management solution designed for high-volume production environments, best suited its needs. Garmin’s operations team feels comfortable leaving AppDynamics on in the production environment, allowing them to respond more quickly and collect historical data. In addition, the product’s usability makes it
a more appealing alternative to some of its competitors: “AppDynamics’ ease of use made it more appealing to developers than other products,” said Strick. “They all preferred the cleaner, simpler approach of AppDynamics.”
Garmin also uses Splunk, a log management system, to aggregate and search their application logs. “I think AppDynamics and Splunk complement each other well,” said Strick. “AppDynamics allows me to capture code execution details without modifying the code for logging purposes.”
Since they started using it, AppDynamics has helped Garmin address several issues in its most important applications. Here are a few of the use cases for AppDynamics at Garmin.
Use Case #1: Memory Monitoring
Before they started using AppDynamics, Garmin was unable to track application memory usage over time.This made it difficult to determine what resources the application would need in coming months and to diagnose issues like memory leaks. Occasionally memory-related issues would degrade the performance of the application and affect Garmin’s end users.
One Sunday, Garmin’s tier 1 support team received reports of slow responses and timeouts on the primary eCommerce application.Tier 2 support was able to bypass the issue by recycling application servers, but the issue reappeared on Monday morning. Because Garmin’s support teams had little visibility into how the application used memory, they could not diagnose the problem and issue a permanent fix.
With the historical data in AppDynamics, Garmin was able to see that the application was using more and more memory over time. In the short term, they were able to add more memory to the VMs and increase the heap size in order to relieve the support teams. In the longer term, they decided to move the cache out of the heap in order to prevent the problem from reappearing. In addition, they set up alerts in AppDynamics on heap usage and garbage collection times so that they could respond more quickly the next time the application began to run out of memory.
Use Case #2: Automated Remediation Using Application Run Book Automation
AppDynamics’ Application Run Book Automation feature allows organizations to trigger certain tasks, such as restarting a JVM or running a script, with thresholds in AppDynamics. Garmin was able to use this feature very effectively one weekend while waiting for a fix from a third party.
After the launch of a new application, a bug was found in a piece of third party code that was preventing customer emails from reaching customer support agents.The problem was that all the application objects were being consumed at once, causing the heap on the JVM to be exhausted. Restarting the JVM would temporarily resolve the issue, but the issue was discovered on a Friday and the vendor would not be able to issue a patch until the next week.
Facing the prospect of manually restarting JVMs all weekend, Garmin’s operations team decided to set up Application Run Book Automation in AppDynamics to automatically restart the JVM when the percent of heap used exceeded a certain threshold.They created a simple Python script, which they invoked with a shell script that was called by AppDynamics.The whole process took less than 30 minutes.They then validated that the restart was successful and then were able to enjoy the weekend.
Use Case #3: Code-level Diagnostics
With AppDynamics, Garmin’s application teams have been able to rapidly resolve several issues originating in the application code.
Customers reported getting an alert on the landing page for map updates, software updates and service alerts.This resulted in a poor experience for customers trying to update many devices through the account management site.
Garmin’s application teams were able to quickly identify the problem in their code by drilling down into an error with AppDynamics. They resolved the issue and now the site performs better for customers with a large number of devices.
Use Case #4: Bad Configuration in Production
Garmin also uses AppDynamics to verify that new code releases perform well. Because Garmin has an agile release schedule, new code releases are very frequent (every 2 weeks) and it’s important to ensure that each release goes smoothly. One common problem that oc- curs when deploying new code is that configurations for test are not updated for the produc- tion environment, resulting in performance issues.
A particular application was experiencing problems after a new code release, which resulted in a bad customer experience.
By looking at the application flowmap in AppDynamics, Garmin’s application teams found that the production application was calling out to a service used in the test environment.
After drilling down into a stalled transaction they discovered that the problem was due to a configuration file not being updated for the production environment. “We would have had no way of knowing this before,” said Strick. “The system dependency dashboard in AppDynamics helped us to quickly identify and resolve this issue.”
Use Case #5: Real-time Business Metrics
With Real-Time Business Metrics from AppDynamics, organizations like Garmin can collect and monitor transaction payload data from their applications. Strick used this capability to measure the revenue flow in Garmin’s eCommerce application.
Garmin didn’t have any way to know eCommerce sales in real time.They wanted a way to show executives how much revenue came through the eCommerce application at any given time, and to use historical data to make informed decisions about how to spend IT resources.
Garmin used AppDynamics to collect and aggregate the dollar value associated with each credit card transaction on Garmin’s eCommerce application.They then monitored this value over time and displayed it in a dashboard that they could present to Garmin’s executive team. “This historical metric data is valuable in providing proof when changes need to be made,” said Strick.
Garmin believes that AppDynamics has proved its value as a production monitoring and troubleshooting tool for their applications. “AppDynamics has given us visibility into areas we never had visibility into before,” Strick said. “Some issues that used to take several hours now take under 30 minutes to resolve, or a matter of seconds if we’re using Run Book Automation.”
In addition, the data in AppDynamics has allowed Garmin to make important decisions about how to allocate resources in the future. “The biggest plus for me has been the historical metric data we have now,” Strick said. “We can now understand how the application is growing over time.This data will prove invaluable in guiding future decisions in IT.”