APM Buyer's Guide

Seven Key Questions to Ask APM Vendors

At AppDynamics, we focus on listening to our customers and their needs. We’ve also noticed their surprise and delight when they see our technology in action and realize they can defy the conventional wisdom of how Application Performance Management can work.

Over the last year, we’ve collected a short list of key questions that APM buyers should ask to ensure they get the solution that best fits their needs. For example, some products like profilers are best suited for development environments, but are a bad fit for production monitoring. The same distinction is true of APM products.

Many APM vendor solutions will work for you if you have a relatively simple environment. However, if your environment has any of the characteristics below, you’d be well served to dig deep into vendor capabilities:

You want a Production environment monitoring solution vs. Dev/Test application profiler

Production Overhead. Many APM products create unacceptable Overhead in Production if you want 100% class/method visibility.

Your application has a highly-distributed or Service-Oriented Architecture (SOA)

Fit for Distributed Application. Diagnosing the root cause of an application problem may be slow or impossible if the APM product cannot continuously track and follow a distributed business transaction across the tiers.

You follow an agile development philosophy with multiple releases a year

Fit for Agile Environment. Many APM products require you to manually instrument new code introduced by agile releases. This can be a costly and time-consuming process.

Your environment is dynamic. Change is the norm. Your user load patterns vary and your application environment changes.

Behavior Learning. If your environment changes frequently, any problem detection approach that is based on static settings and manual thresholds will generate false alarms.

Your application leverages modern approaches like distributed caches, ORMs, or NoSQL databases

Modern Diagnostics. Many APM products may have blind spots in these areas and won’t help you troubleshoot application problems.

You have a lot on your plate already and want this to be easy. You don’t have the personnel or consulting budget to dedicate someone to managing the APM product.

Easy-to-Deploy & Use. Many APM products require expensive consultants or dedicated staff to properly configure the instrumentation, dashboards, and alerting.

Your budget for Application Monitoring

Low Cost of Ownership. There can be a 5x difference in the the hardware, software and people costs for different APM solutions.

 

Here is a list of recommended questions to ask APM vendors:
  1. Fit for Production Environment
    • What is the overhead in production if I want visibility into every class & method?
    • When setting up the system, am I forced to make compromises on level of visibility to be able to have an acceptable level of overhead? Why am I forced to make that compromise?
    • Do you recommend that users instrument every class & method? If not, why not?Can the system scale to support thousands of reporting agents to a single management server?
  2. Fit for Highly-Distributed or SOA environments
    • Has your user interface been designed for users to monitor the health of individual Application Servers or does it enable the monitoring of distributed Business Transactions?
    • How does the product automatically discover and visualize the relationship between services and the back-end tiers? Can it measure and display the latency of each hop?
    • What technology approach is used to track & follow business transactions across the distributed environment and provide code-level diagnostics at each tier?
    • What percent of my business transactions will it discover out-of-the box?
    • Can the product identify communication problems between the tiers/services in a SOA environment?
  3. Fit for Agile Environments
    • Does your product require me to manually instrument the code? If so, how long does this take per application?
    • When agile code is introduced, does the instrumentation have to be manually updated? If so, how much time does this typically take your clients?
    • Does your product allow me to easily analyze the how a new release may have degraded application performance and pinpoint the root cause down to the code level?
  4. Behavior Learning
    • Does your product continuously learn the normal performance of the application or do I have to set manual thresholds?
    • Can it learn different load and response time patterns for different days or time of day?
    • Can it learn "normal" deviation patterns and take action automatically based on what it observes?
    • Does the system automatically adapt to dynamic changes in capacity in a virtualized or elastic computing environment?
  5. Reduce Mean-time-to-Resolution (MTTR) with Troubleshooting & Root Cause Diagnostics
    • When a problem occurs, is diagnostic data automatically captured or do I have to manually trigger a diagnostic session?
    • Are there code-level diagnostic automatically captured for every bad session?
    • How does the product distinguish one-time anomalies from consistent patterns of poor performance? What actions does the product take when it detects systemic failures?
    • Does the product provide specific diagnostics for each of the following:
        • Slow response
        • Errors
        • Stalls
        • Deadlocks
        • Malformed code
        • Slow SQL/DB access
        • Memory Leaks & Thrash
        • Distributed service communication issues
        • Synchronization issues
  6. Easy-to-Deploy & Use
    • Can I deploy the product myself in less than an hour? Or do you recommend that I have your consultant assist me in the deployment?
    • Does your product user interface require experts to interpret the data when problems happen and rely on them to determine root cause? Or, does the product pinpoint root cause automatically?
    • Does your product have an intuitive design that I can I access your product from a web browser?
  7. Total Cost of Ownership
    • What is the initial software license cost?
    • What is the total cost of hardware to support the APM solution deployment? How many collectors? How many Managers of Managers?
    • What is the annual labor and consulting costs for APM Product Configuration / Instrumentation?
    • What are the costs for ongoing maintenance, application of software upgrades of the APM solution?

This is just a short list. It probably won’t make us very popular with legacy APM vendors, but we’re here to make you successful, not them.

If you’d like access to our full APM Evaluation Guide, please contact AppDynamics at info@appdynamics.com