AppD on AppD: Scaling Our Custom Dashboards Platform

This blog post is a developer’s perspective on how using our own AppDynamics software has helped us find and fix performance-related issues – and how other developers can do the same. 

One of the most challenging aspects of developing cloud-based platforms is scalability. As we innovate and build new features, it is essential for developers to ensure these new platform features are scalable and do not impact the performance of our applications, especially as any impact on performance is likely to impact all tenants on the cluster.

This is a challenge that we experienced firsthand. My team here at Appdynamics is responsible for building custom dashboards, one of the most powerful features of AppDynamics. It allows you to group relevant metrics into one central dashboard as well as build sophisticated dashboards with drill down capabilities.

As we added more capabilities and support for the different types of metrics, we realized that scaling our dashboards was becoming a challenge. Many of our customers were exploring different dashboard capabilities and features, which was truly stressing our endpoints. This is when our practice of using AppDynamics internally – “AppD on AppD” – helped give us insight and visibility into the issues we faced while developing new features on the platform.

Our AppD on AppD Approach

Recently, we made some changes on the backend of a REST endpoint that returns metrics for different types of widgets on custom dashboards. Our team’s internal process is to ensure every feature we implement is stress-tested under a real type of load profile, so we made sure to test this as well. Below is what we discovered.

On local dev environments, the feature worked fine and the data returned by the API took a few milliseconds. Next, we deployed our changes to our performance environment, which is similar to an actual production instance with large amounts of data. We immediately noticed that under stress, the UI which uses the endpoint was very slow and the average response time of the endpoint was considerably high. Luckily, since AppDynamics was monitoring the performance environment, it was easy for us to dig into the issue.

We configured a business transaction to monitor the REST endpoint with the slow response time and within a few minutes, we collected transaction snapshots which gave us valuable behind-the-scenes information of the REST API calls. There were two things that caught our attention:

Resolved Issue #1

A particular method which was being called repeatedly was taking more time to execute than before. The method itself took around 100-150 milliseconds, but if it was called 100 times in a single transaction, it would take around 15,000 milliseconds to execute, which is roughly 15 seconds.

The image below shows the total time it took for all the calls of this method in this single transaction.

Here is a code snippet of that method:

After a few minutes of looking at the full implementation of this method, we found that String.replace could be a potential problem here, which happens to be slower than StringUtils.replace. As a result, we made a minor change and modified the code to use StringUtil implementation. Here is some information on StringUtils.replace vs String.replace.

Resolved Issue #2

Another issue we noticed was that there were too many database calls for a single request.

We made a few optimizations here, including caching the data and bulking-up the queries. After applying these fixes, we measured the performance again and saw the average response time for the endpoint improve drastically.

Lastly, we created our own custom dashboard to measure performance of our specific endpoint, showing us different metrics like average response time, errors, and calls per minute along with thresholds and baselines. We also created a scheduled report that sends our team a dashboard snapshot everyday, allowing us to easily spot outliers and proactively address issues. We also could have created alerts, which notify you when certain conditions are met or exceeded based on your configuration. Scheduled reports and alerts make it easy to spot outliers and proactively address issues.

Without AppDynamics, it would have been difficult for a developer to quickly pinpoint these issues within a large codebase. But with AppDynamics, not only does it become easier to find issues in production, but developers can proactively ensure that features are robust and scalable before deploying them to production. This reduces the time spent on performance-related issues, and instead, gives developers more time to innovate and write code.

Learn more about our business-centric dashboards or get a demo from our sales team today.

How to Rev Up Your Business Performance During the Holiday Season

Both buyers and businesses love Black Friday and Cyber Monday. The online retail demand however is not limited to just those two days. It often stretches from days prior to Black Friday and after Christmas.

This year, Forrester predicts that online holiday sales in the United States will reach $129 billion in 2017 – a 12% growth in holiday sales over 2016. And this shouldn’t come as a surprise, as consumers continue to spend more online due to the convenience, selection, and value.

As a result, online retailers understand the importance of delivering a seamless digital experience to their customers. But unfortunately, technical issues have plagued retailers during this peak season, likely costing them millions of dollars in sales.

For example, last year, Macy’s website went down the morning of Black Friday and was experiencing outages through the afternoon. As a result, most users were greeted with a “temporary shopping jam” message due to overflow shopping traffic.

The year before, Neiman Marcus and Target experienced similar issues.

With some of the biggest retailers struggling to maintain a seamless digital experience for their customers during the holiday season, many businesses are now asking themselves the following questions:

– How can you offer an improved digital experience while ensuring your website meets the sustained and peak demand the holiday season is bound to usher in?

– How do you know if those coupons you are offering will ultimately result in increased sales?

– How can you track user journeys to analyze in real-time what the impact on the business is and where are the drop-offs?

– How do you take all of the above and tie it back to a problem which may be related to your application or infrastructure to fix them in real-time?

The answers to these questions can be found with AppDynamics Business iQ. An earlier post explored how you can leverage AppDynamics Business iQ to analyze how application performance is impacting customer experience and business outcomes. Business iQ allows developers, ops leaders, and marketers and executives to share a common interface for reviewing the health and performance of the business in real-time.

For example, Nasdaq, uses Business iQ to understand usage trends of their applications. Dhisco, an online payment company, segments its largest hotel chain customers to measure critical metrics such as total bookings and room availability across its major customers. And apparel company, Carhartt, uses Business iQ to monitor its purchase funnels to quickly identify application performance issues and how they may be impacting conversion rates.

How Carhartt Leverages Business iQ on Cyber Monday

Below is an excerpt from a Carhartt Case Study.

Established more than 128 years ago, Carhartt created the first purpose-built bib overalls for railroad workers. John Hill, Carhartt’s chief information officer, says that as Carhartt expanded its global footprint, it was important for IT to deliver a core set of capabilities that could scale as needed to different regions, rather than having different infrastructures and different application capabilities for every market.

To help with their digital transformation, Carhartt enlisted AppDynamics for real-time performance monitoring. Carhartt’s IT team had been relying on an alerting system to identify issues in four on-premises environments. But the system didn’t meet the team’s need to react in real time to changes in applications or the market. And the information it provided didn’t overlap with business needs.

Hill knew it would take time to achieve end-to-end visibility across Carhartt’s entire ecosystem, but he also believed he could realize immediate value from AppDynamics by focusing on specific processes like ordering, inventory, and checkout.

Leading up to Cyber Monday 2016, the team built a dashboard that tracked sales velocity and response times. “We wanted to see what was happening in real time as orders were coming in and how the infrastructure was supporting those orders,” Hill says.

On Cyber Monday, the engineering team was settled into their war room to monitor the action, where they were later joined by members of the senior management team. It turned out Hill had sent out a link to the new dashboard to the leadership. For the first time, Carhartt’s executives could watch orders being placed and processed. “The president of the company joked that the Business iQ dashboard was addictive,” Hill says.

As business leaders looked on, the engineering team spotted an issue in the order processing system that could have had a direct impact on the day’s revenue. The issue was fixed in ten minutes. “Before deploying AppDynamics and Business iQ, we wouldn’t have known about the problem until customers started complaining to the help desk,” Hill says. “We were able to quickly readjust and resolve the problem without our customers knowing that a problem was starting to occur.”

In addition, Hill notes that the real-time capabilities of Business iQ changed the relationship IT had with their business colleagues. The biggest change for his team was the ability to be nimble and respond quickly to the business needs in a high pressure situation like Cyber Monday and create real impact for the business.

Read the full Carhartt Case Study to see AppDynamics BiQ enabled Carhartt to have a seamless Cyber Monday that registered more than a record million dollars in sales.

To learn more about AppDynamics, check out The AppPerformers, a compilation of 50 short case studies of how enterprises have leveraged AppDynamics to improve their application and business performance.

Four Use Cases for Leveraging Business iQ

The application and the business have converged.

In fact, the performance of your business is now inseparable from the performance of your apps. Customers who are connected to the code you write or the applications and infrastructure you manage demand a flawless experience, and they are loyal to the brand that delivers it. The challenge, however, is that the traditional ways of managing services and operations are falling short, jeopardizing business success.

Many BI tools today help analyze business data, but they are mostly for historical analysis and trends. Web analytics tools on the other hand help analyze customer conversion rates and how end-users are using your website, but don’t tell you why it’s happening. And then there are traditional APM tools that tell you whether your applications are healthy or performing poorly, but offer little visibility into the impact on your business. As a result, many organizations are struggling to connect data to business outcomes.

Business iQ offers real-time context from customer to code – connecting application performance data to business outcomes to enable both the business and IT.

To see how you can leverage AppDynamics Business iQ, check out these four common use cases.

Business Health

Let’s take a scenario where there is a major event, like Black Friday or a launch day for an important product, or simply any day you need to win customers. If you are using AppDynamics for your APM solution, your DevOps may receive alerts about anomalies in your application’s performance. When this happens, business owners may want to understand the impact on key business drivers, as well as any revenue and customer experience implications. With Business iQ, you can monitor critical business KPIs to get real-time insight into the health of your business.

Figure 1

Using Figure 1 (above), let’s say you are an e-commerce site and manage a number of different brands. Overall, you care about your site’s conversion rate, number of orders processed, total sales, and the percentage of customers moving to your loyalty program.

It can be inferred from the dashboard that in the last hour, ‘Total Sales’ may be declining since not many of your loyal customers are shopping today. What’s more, you can also see from the dashboard that there’s a clear correlation between the drop in sales revenue and the the % errors in APDY Electronics. However, with just a 1% error in APDY Books, there may be a business problem impacting sales versus an IT issue.

With this level of insight, DevOps teams will be able to investigate root-cause for the performance issue, or declare that it’s not an IT problem. Business health monitoring attempts to converge business data with application and infrastructure data to give you visibility into business KPIs that allow you to diagnose and fix problems in real time.

User Journey

Now that we’ve covered business health monitoring, let’s move on to our second use case: User journey monitoring, which measures how business components and customer experience come together to drive top-level KPIs. Figure 2 (below) helps illustrates this use case:

Figure 2

Let’s say you’re a bank that would like to understand how your users are moving along the loan processing journey – from viewing rates online, submitting an application on your website, running a credit check, and finally getting the loan approval.

Business teams might be interested in knowing how many customers are in the loan journey, where there are drop-offs, and how KPIs are impacted. Your operations team, however, may be curious if customer drop-offs are a result of slow application performance. And lastly, from a developer standpoint, you may want to know how you are impacting a larger process and causing real-business impact.

In the above dashboard, you can see that a longer response time during the “Submit Application” step at 7s, is probably causing a higher drop-off, and therefore impacting the loan amount processed at $10M. Furthermore, a 15% error rate at “Credit Check” is further compounding this problem at the “Loan Approval” stage.

User Journey Monitoring allows you to visualize different parts of a process, driving a common language between business and IT, whether you are a bank looking to optimize your processes or a retailer trying to visualize how your customers shop online.

Customer Segment

Interested in understanding the end-to-end experience of your critical customer segments? With Business iQ, you can. Check out the use-case below for customer segment monitoring.

Figure 3

Let’s say you’re a travel company that connects back-end travel inventory (think flights, hotels, etc.) to multiple front-end buying channels (think websites used to book travel, like Expedia, Orbitz, Priceline, etc.) With Business iQ, you can compare customer experience across these various buying channels, and segment customers based on error codes, slow transactions, and more. You can also analyze customer experience across various travel agents or hotel brands.

This deep-level end-user monitoring is critical for businesses, as it allows you to monitor customer issues so you can proactively avoid them in the future.

Your DevOps team might also be interested in customer segment monitoring to understand where to prioritize troubleshooting efforts. For example, by looking at the above dashboard, it’s clear that Priceline customers are experiencing issues with reservation confirmation and search availability, and that the “5 Star Luxury” hotel class segment has the highest error reservations. This information allows DevOps teams to have visibility into how their critical customer segments are performing and prioritize what they want to work on.

This capability allows your business to get visibility into how application performance impacts customer interaction with your features.

Release Validation

You can also use AppDynamics in real-time as you release newer versions of your application, or migrate from a legacy infrastructure to a new infrastructure. Business iQ also allows you to compare KPIs from your previous and current versions.

Figure 4

Let’s consider this example using Figure 4 (above): APDY media has an entertainment site where customers sign up for subscriptions. The journey they go through starts with creating a profile, selecting their favorite content, setting alerts on what content they want to be notified about, and finally, subscription confirmation.

Looking at the above dashboard, it is evident that in Version 1.0 of APDY media, the subscription rate starts to decline significantly, and it appears to be linked to a performance issue when selecting “Favorite Content.”

You can use Business iQ to help identify the problem and understand where to fix the issue. You can also use Business iQ to compare releases and identify if the problems you wanted to fix in your latest release are indeed fixed. You can do this by seeing if performance in the “Favorite Content” step improved post-release and how it correlates to an increase in conversion.

You can also see from this dashboard that while you are sending a smaller amount of traffic to Version 2, you still managed to triple your conversion rate while also driving much higher subscriptions.

The above use cases are just some of the common scenarios we are seeing in our customer environments. As you try our product, you’ll learn that there’s a lot more you can achieve with Business iQ.

Harish Doddala is leading product growth and adoption initiatives of Business iQ at AppDynamics. He has over 10 years of Product Management and Software Engineering experience delivering results for Cisco, VMware, and Oracle.

United Airlines and Real-time Business Monitoring

Flying out in front

One of the world’s largest airlines, United Airlines, has an impressive history of using one of humankind’s greatest innovations — the airplane — to serve people. On April 6, 1926, pilot Leon “Lee” Cuddeback flew a Laird Swallow biplane flew from Boise, Idaho, to the railroad mail hub at Pasco, Washington, returning the following day with 200 pounds of mail. This historic round trip journey was the first contract airmail flight, making United Airlines (then called Varney Air Lines) the first ever commercial airline in the U.S. to fly fixed routes and schedules. Another customer-centric first was established in 1930, when United hired nurse and pilot Ellen Church to assist passengers and help calm them of their fear of flying, making her the first female flight attendant on a passenger airplane.

Millions of flights and billions of passengers later (140 million in 2015 alone), United is more dedicated than ever to using innovation to keep the skies, terminals, and maintenance hangars friendly for their passengers and employees.

“One of the goals given to us by leadership was, let’s win back the trust of employees and our customers,” says Bill Hineline, United’s Director of Application Performance Management.

No small task, seeing how Mr. Hineline’s team is responsible for managing over 125 applications, from and ticketing to weight balance systems and baggage tracking.

United has enjoyed what has become a truly special partnership with AppDynamics, including Mr. Hineline taking the stage at the AppSphere 2016 keynote and leading a session where he revealed the depth of the successful partnership, which now includes Business iQ, our real-time business monitoring solution.

“We now understand business impact,” says Mr. Hineline. “Who is using it? How are they using it? Where are their problems being experienced?”

We couldn’t be prouder of the relationship and how we’ve been able to assist United in their digital transformation, allowing them to continue on the path of constantly innovating to create happy travelers and team members — and drive better business outcomes.

Bill Hineline will be joining us again to talk about how AppDynamics and United are working together to better serve customers, employees, and the business during an upcoming webinar that you won’t want to miss. Register now.

Want to hear more about the United-AppDynamics partnership? Watch the inspiring story here.


Real-time Business Monitoring with Business iQ

Business iQ is the revolutionary new App iQ performance engine that enables business outcome monitoring in addition to your traditional application monitoring. AppDynamics’ unique and patented approach to application performance management (APM) has allowed us to analyze business metrics to better understand your application’s impact on your bottom line. As more and more businesses tackle digital transformation, the pressure to move at a high velocity increases, and managing the risk of each iteration becomes very important. Did your new release improve customer satisfaction? Are more customers doing self-service online successfully? Have profits increased? These are all questions that need to be answered in real-time to support a low “Mean Time to Business Awareness” (MTBA) that will fuel your digital transformation.

The key to offering real-time business monitoring with Business iQ lies in the careful integration of all of AppDynamics’ iQ engines, resulting in a dynamic, accessible set of data via Signal iQ. With our ability to tag and trace every business transaction — from a user’s device all the way through their entire app experience — AppDynamics is able to see every customer journey and interaction. We gather the business and performance data provided from each session into Signal iQ. Business iQ is then able to extract that information and generate baselines, dashboards, and give you perspective beyond traditional APM. You can now observe conversions, engagement, and other business outcomes. AppDynamics’ unique design for scale allows our agents to live within the production environment, giving us the ability to display information in real-time, directly from the heartbeat of the application.

When the data arrives from moment to moment at the Signal iQ repository, you can immediately query it using ADQL, an SQL-based query language developed by AppDynamics. ADQL allows for dashboarding or creating automated data collectors that aggregate specific metrics to intelligently baseline using our Baseline iQ engine. Intelligent baselining lets business owners and stakeholders to have real-time views of business performance. Baseline comparison allows for the detection of anomalies in the business performance that could later be attributed to the application performance, UI changes, or other external factors.

Via Map iQ, the App iQ platform allows you to map every user interaction to a business transaction, baseline every metric, and intelligently collect information through Baseline iQ. Diagnostic iQ provides session data snapshots when performance deviates from baseline using. With the addition of Business iQ, you’ll now be able to extend application monitoring into business monitoring and develop a common language to minimize risk and accelerate your digital transformation.


Find out more about Business iQ

Accelerate the Velocity of Your Business

Businesses today need to move at high velocity in order to remain competitive. The digital transformation that has propelled many industries to make investments in their applications now requires tighter alignment between business and IT.

In order to achieve ideal velocity, companies must change their culture from a traditional waterfall software delivery system to one of continuous delivery in order to innovate more quickly. The days of having large-scale releases are now gone and businesses must drive new feature launches in small, incremental upgrades. However, this new approach also presents some challenges for the business as well as IT. Understanding the health of an application instantly after every release is important to assess its quality, and recognizing its relationship to the health of the business and the customer experience just as quickly is crucial to running a successful and agile enterprise.

Having the right monitoring platform in place becomes critical to managing a business that’s able to innovate continuously. This system must cross from IT and DevOps to the business side of things to provide an understanding of how performance affects your financial objectives. The tension that previously existed between business and IT that led to internal battles stands in the way of innovation. According to Forrester’s Report, CIOs: Use Your BT Agenda To Support eBusiness Strategy, “Achievement can be accelerated with shared goals between IT, development, and management.”

In order to maintain a competitive advantage, enterprises must embrace high velocity innovation by leveraging better collaboration with IT and business to understand the connection between application performance and the bottom line in real time.