How Top Investment Banks Accelerate Transaction Time and Avoid Performance Bottlenecks

A complex series of interactions must take place for an investment bank to process a single trade. From the moment it’s placed by a buyer, an order is received by front-office traders and passed through to middle- and back-office systems that conduct risk management checks, matchmaking, clearing and settlement. Then the buyer receives the securities and the seller the corresponding cash. Once complete, the trade is sent to regularity reporting, which insures the transaction was processed under the right regulatory requirements. One AppDynamics customer, a major financial firm, utilizes thousands of microservices to complete this highly complex task countless times throughout the day.

To expedite this process, banks have implemented straight-through processing (STP), an initiative that allows electronically entered information to move between parties in the settlement process without manual intervention. But one of the banks’ biggest concerns with STP is the difficulty of following trades in real-time. When trades get stuck, manual intervention is needed, often impacting service level agreements (SLAs) and even trade reconciliation processes. One investment firm, for instance, told AppDynamics that approximately 20% of its trades needed manual input to complete what should have been a fully automated process—a bottleneck that added significant overhead and resource requirements. And with trade volumes increasing 25% year over year, the company needed a fresh approach to help manage its rapid growth.

AppDynamics’ Business Transactions (BT) enabled the firm to track and follow trades in real time, end-to-end through its systems. The BT traces through of all the necessary systems and microservices—applications, databases, third-party APIs, web services, and so on—needed to process and respond to a request. In investment banking, a BT may include everything from placing an order, completing risk checks or calculations, booking and confirming different types of trades, and even post-trade actions such as clearing, settlement and regularity reporting.

The AppDynamics Business Journey takes this one step further by following a transaction across multiple BTs; for example, following an individual trade from order through capture and then to downstream reporting. The Business Journey provides true end-to-end, time-enabling tracking against SLAs, and traces the transaction across each step to monitor performance and ensure completion.

Once created, the Business Journey allows you to visualise key metrics with out-of-the-box dashboards.

Real-Time Tracking with Dashboards

Prior to AppDynamics, one investment bank struggled to track trades in real time. They were doing direct queries on the database to find out how many trades had made it downstream to the reporting database. This method was slow and inefficient, requiring employees to create and share small Excel dashboards, which lacked real-time trade information. AppDynamics APM dashboards, by comparison, enabled them to get a real-time, high-level overview of the health and performance of their system.

After installing AppDynamics, the investment bank instrumented a dashboard to show all the trades entering its post-trade system throughout the day. This capability proved hugely beneficial in helping the firm monitor trading spikes and ensure it was meeting its SLAs. And Business IQ performance monitoring made it possible to slice and dice massive volumes of incoming trades to gain real-time insights into where the transactions were coming from (i.e., which source system), their value, whether they met the SLAs, and which ones failed to process. Additionally, AppDynamics Experience Level Management provided the ability to report compliance against specific processing times.

Now the bank could automate complex processes and remove inefficient manual systems. Prior to AppDynamics, there was a team dedicated to overseeing more than 200 microservices. They had to determine why a particular trade failed, and then pass that information onto the relevant business teams for follow-up to avoid losing business. But too often a third-party source would send invalid data, or update its software and send a trade in an updated format unfamiliar to the bank’s backend system, creating a logistical mess too complex for one human to manage. With Business IQ, the bank was able to immediately spot and follow up on invalid trades.

Searching for Trades Across All Applications

Microservices offer many advantages but can bring added complexity as well. The investment bank had hundreds of microservices but lacked a fast and efficient way to search for an individual trade. In the event of a problem, they would take the trade ID and look into the log files of multiple microservices. On average, they had to open up some 40 different log files to locate a problem. And although the firm had an experienced support staff that knew the applications well, this manual process wasn’t sustainable as newer, inexperienced support people were brought onboard. Nor would this system scale as trade volume increased.

By using Business IQ to monitor every transaction across all microservices, the bank was able to easily monitor individual trade transactions throughout the lifecycle. And by capturing the trade ID, as well as supplementary data such as the source, client, value and currency, they could then go into AppDynamics Application Analytics and very quickly identify specific transactions. For example, they could enter the trade ID and see every transaction for the trade across the entire system.

This feature was particularly loved by the support staff, which now had immediate access to all of a trade’s interactions within a single screen, as well as the ability to easily drill down to find the find the root cause of a failed transaction.

Tracking Regulatory SLAs in Real Time

Prior to AppDynamics, our customer didn’t have an easy way to track the progress of a trade in real time. Rather, they were manually verifying that trades were successfully being sent to regulatory reporting systems, as well as ensuring that this was completed within the required timeframe. This was difficult to do in real time, meaning that when there was an issue, often it was not found until after the SLA had been breached. With AppDynamics they were able to set up a dashboard to visualise data in real time; the team then set up a health rule to indicate if trade reporting times were approaching the SLA. They also configured an alert that enabled them to proactively see and resolve any issues ahead of an SLA breach.

Proactively Tracking Performance after Code Releases

The bank periodically introduces new functionality to meet the latest business or regulatory requirements, in particular MiFID II, introduced to improve investor protection across Europe by harmonizing the rules for all firms with EU clients. Currently, new releases happen every week, but this rate will continue to increase. These new code releases introduce risk, as previous releases have either had a negative impact on system performance or have introduced new defects. In one two-month period, for instance, the time required to capture a trade increased by about 20%. If this continued, the bank would have had to scale out hugely—buying new hardware at significant cost—to avoid breaching its regulatory SLA.

The solution was to create a comparative dashboard in AppDynamics that showed critical Business Transactions and how they were being changed between releases (response times, errors, and so on). If any metric degraded from the previous version or deviated from a certain threshold, it would be highlighted on the dashboard in a different color, making it easier to decide whether to proceed with a rollout or determine which new feature or change had caused the deviation.

Preventing New Hardware Purchases

After refining its code based on AppDynamics’ insights, the bank saw a dramatic 6X performance improvement. This saved them from having to—in their words—“throw more hardware at the problem” by buying more CPU processing power to push through more trades.

By instrumenting their back office systems with AppDynamics, the bank gained deep insights that enabled them to refine their code. For instance, calls to third-party APIs were taking place unnecessarily and trades were being captured unintentionally within multiple different databases. Without AppDynamics, it’s unlikely this would have been discovered. The insight enabled the bank to make some very simple changes to fine-tune code, resulting in a significant performance improvement and enabling the bank to save money by scaling with their existing hardware profile.

Beneficial Business Outcomes

From the bank’s perspective, one of the greatest gains of going with AppDynamics was the ability to follow a trade through its many complex services, from the moment an order is placed, through to capture and down to regularity reporting. This enabled them to improve system performance, avoid expensive (and unnecessary) hardware upgrades, quickly search for trade IDs to locate and find the root cause of issues, and proactively manage SLAs.

See how AppDynamics can help your own business achieve positive outcomes.

Improved Database Support Enhances Data Visibility and Helps Resolve Problems Faster

Total visibility is crucial to monitoring the database environments that support your most critical applications. That’s why AppDynamics recently released a number of updates to our Database Visibility functionality. Not only do we now support monitoring Couchbase, but we also added snapshot correlation, support for custom cluster configuration, and enhancements to our ability to collect and track custom database metrics. These updates provide even more ways for you to get the insights you need to take action.

Couchbase DB Support

NoSQL databases can be a critical component when developing modern applications that need to deliver simple, high performance data transactions to provide the best possible customer experience. AppDynamics has added support for Couchbase DB in an effort to provide comprehensive monitoring for the entire life of a business transaction including at the database level. You can use AppDynamics Database Visibility to remotely monitor your Couchbase clusters, without any component or code modifications. Monitor N1QL queries on all database activity and review historical time-series graphs of time spent in Couchbase (supplemented by underlying database-specific metrics) to proactively identify and resolve performance issues.

Snapshot Correlation

Sometimes it can be difficult to see how database performance is affecting your overall business transactions. With snapshot correlation, we can match and tag business transactions and snapshot identifiers to database visibility data to provide a better integration across your entire solution. The new Database Queries tab on the Business Transaction dashboard displays correlated queries and database data directly in your APM Dashboard for a more comprehensive dataset.

Custom Cluster Support

Database clustering technology is not implemented consistently across all database platforms. This can make it difficult to monitor an Oracle RAC cluster in the same way that you monitor a sharded MySQL database. We now provide enhanced support for database clusters, including Master/Slave technologies and sharded architectures. You can define a single collector configuration to monitor multiple database instances and treat them like one entity, allowing for a better visualization within the UI.

Custom Query Metric Enhancements

At AppDynamics, we believe your data should work for you instead of the other way around. To that end, we have introduced some improvements to the way we handle custom queries and metrics for Database Visibility. You can now scale your custom queries for deeper insight into your database with several new features, like scheduling and bulk add activities. Using the new Custom Metrics tab in the Dashboard view, you’ll see data trends faster without having to create custom dashboards. Test custom queries before creating them, saving you time debugging errors later. These are just a few of the recent additions!

How to Get Started

If you want to start using these new features, be sure to upgrade your Database Agents to 4.4.4+ and upgrade your controller software to 4.5+. If you are not yet using Database Visibility to monitor your databases you can get started by reading our documentation.

Visit our product updates page to stay informed on all our recent product developments.

The AppD Approach: Database Monitoring with Service Endpoints

I keep running into cases where the 80/20 rule (also known as the Pareto Principle) helps me prioritize my efforts in APM. I find that 80% of my problems are easy to fix, and once they are fixed, they stay that way. The other 20% require constant attention. It’s even tougher when one of the 20% is mission critical. Some examples might be a database query that touches multiple tables and can be de-optimized by changes in schema, indexing or even table row counts. Or, outside of database monitoring, the issue may lie in a key sequence of code that is regularly touched by developers, and it keeps breaking as they perfect their understanding of the problem. If you are running into these kinds of scenarios, it might make sense to continuously watch the performance of your queries (or code) and have a handy way to compare what your queries were doing in the past to what’s happening in the present moment. You might also want to create some health rules so AppD will tell you right away if performance has degraded. In this blog post, I’ll discuss how to do that using a Service Endpoint.

The first step is to get a list of snapshots from the last time the problem cropped up. For database monitoring, the “Slow Response Times” screen is good place to start.

To get there:

1.     Expand the “Troubleshoot” item in the left side navigation bar

2.     Select “Slow Response Times” in the left side navigation bar

3.     Select the “Slowest DB & Remote Query Times” tab

4.     Filter on the JDBC type calls

5.     Select a query from the list and click “View Snapshots”

Select a snapshot that has full call graph data like this one.

Find the snapshot segment that makes the query call, and the find the method in the call graph that made the query, right click on it, and select “Configure Instrumentation for this Class/Method”.

In the next dialog choose “Service Endpoint”. Provide a name for this endpoint in the Service Endpoint Definition dialog.

If you were lucky enough to have a single class and method in your Data Access Object (DAO) class for each query then you are done. In this case, we have a generic query class that is capable of making any type of query.

AppD can handle that by splitting the Service Endpoints. Back in the Service Endpoint Definition dialog change to the “Transaction Splitting” tab. Enable splitting with the checkbox. Then use the object instance and a getter chain to do the splitting. Enter a getter string like this to access the first part of the query string and clean it up for presentation.

After that, you will be able to track the performance of that query in the Service Endpoints area of the UI, and you can create a health rule too.

These steps will give you a new entry in the Service Endpoints list like this. The Service Endpoint will let you count all the calls into this query and see its Average Response Time and Error Count in real time and in the past [starting from this time forward].

It will also create a new group of metrics in the metric browser that you can use as a data source for a health rule. For example, you may want to create a health rule that fires the next time the average response time of the method tied to your query goes over 100ms (or some other value that is well over the expected response time).

The next time the query causes trouble, AppD will tell you proactively that the query time is slow. The first-level responder will see the health-rule violations in the event list right along with the reports that one or more Business Transactions are slow and impacting your customers. If this information about the query time proves to be really decisive in finding the problem’s root cause, you might even create a policy that sends the health rule violations directly to the team responsible for the query. That will shortcut any root-cause discussion before it even starts.

As I’ve explained in this blog post, Service Endpoints can make make it much easier to monitor mission-critical database queries. As detailed in my previous post, “How to Identify Problematic Database Queries with Business Transactions,” AppDynamics also helps you prioritize the inevitable issues that pop up during database monitoring so you can focus on optimizing code that is most critical to your application. Together Business Transactions and Service Endpoints are two great tools for ensuring the performance of applications.

Find out more about AppDynamics’ database monitoring solutions here.

John Aronson is part of AppDynamics Global Services team, which is dedicated to helping enterprises realize the value of business and application performance monitoring. AppDynamics’ Global Services’ consultants, architects, and project managers are experts in unlocking the cross-stack intelligence needed to improve business outcomes and increase organizational efficiency.

Monitoring ADO.NET Performance

One of the most overlooked parts of .NET is the glue that links your application through to a database known as ADO.NET (an evolution of ActiveX Data Objects). The lowly connection string is the only real access we get to the underlying technology.

I was with a client recently and noticed they had the following connection string:

I was interested in why they had a pool size setting of 1500. The answer lay in the fact that 2 years ago they had a pooling issue (much like the one we’ll explore shortly!) and had set the Max Pool Size to 15 times more than the default!  We then set about seeing if that was needed with their current application.

Using AppDynamics we monitored the calls to their database from the Application’s perspective, you can automatically detect the calls:

Down to the methods where the connection and queries are run:

Extracting the Connection String (without the password) and seeing how long it takes to get a connection from the pool:

These snapshots provide deep visibility into an application’s health and performance, but in the case of performance issues – what’s the easiest way to tune production applications? What happens we you start to see the following?:

ADO.NET pools connections for reuse, there are a number of key rules for ADO.NET and using these connections:

  •       “Open Late and Close Early” – Don’t hold on to connections in code, only use them when you need to.
  •       Beware of “Pool Fragmentation” – Connections are pooled according to the connection string plus the user identity. Therefore, if you use Integrated Security, you may get a Connection pool for every user.

To aid tuning the following parameters can be used:

Using Windows performance counters you can monitor multiple ADO.NET metrics, for any .NET deployments I’d recommend monitoring:

  •       The Number of Pooled Connections
  •       Number of Active Connections

Within AppDynamics, you can monitor and baseline in your production application the health of your ADO.NET connection pool strategy.

The following illustrates a custom dashboard created to simply monitor the database throughput rate and response time against the size of a connection pool. In this scenario by slowly ramping up the number of concurrent users on the application, you can visualise whether your application will scale.

As new users are added to the application, the Connection Pool size increases automatically. This metric can be monitored in conjunction with database response time, error rates, and overall load.

SQL Server, for example, has a default maximum connection pool of 100 and default connection timeout of 30 seconds. I often see organisations dropping the Connection Timeout to 10-15 seconds if you are going to fail to get a database connection it’s better to fail sooner than later, leaving your user hanging for 30 seconds!

Utilize AppDynamics database view to monitor average and maximum pool times:

By registering a performance counter, AppDynamics will automatically baseline and then enable you to configure effective alerts for your application.

The following performance counters are off by default (they can be enabled in the Web.Config file see ActivatingOffByDefault):

  •       NumberOfFreeConnections
  •       NumberOfActiveConnections
  •       SoftDisconnectsPerSecond
  •       SoftConnectsPerSecond

By enabling NumberOfActiveConnections, you can visualize how the connection pool size increases.

To add any performance counter into AppDynamics use the following tool:

Start monitoring your .NET application, check out a FREE trial today!

AppDynamics and MongoDB Work Together to Scale Databases [VIDEO]

If you haven’t noticed, enterprise IT is going through a generational inversion. Not necessarily in people (although that seems to be starting too) but in technology. There is a move from on-premise to cloud. From difficult to use enterprise applications with extended training required, to consumer-grade, intuitive interfaces that are nearly as easy to use as your mobile phone. From the legacy databases that dominated enterprise technology in the 90s and early 2000s for managing relational data types to agile, scalable NoSQL data stores for managing semi-structured, polymorphic, object-style, evolving structured data, and other data shapes.

AppDynamics is leading and learning from this generational shift and our partner, MongoDB, is too. We have seen enterprise IT developers across nearly all verticals — agriculture, financial services, government – leverage MongoDB to better manage their business. That is why in 2011, we built an extension for MongoDB support to plug into our Application Intelligence Platform. This monitoring extension provides visibility into the directory structure and a wealth of metrics on your servers like uptime, flushes, memory utilization, and query counts in addition to stats on your database like object counts and sizes, index counts, and namespace file sizes. But really, what is most interesting about this plugin is the ability to view across the application stack, automatically discover the MongoDB back-end, and identify when you have a problem that exists anywhere in between. Full end-to-end visibility is critical for developing and running modern, highly distributed applications in production.

With the continued rise of NoSQL databases in general, and MongoDB specifically, we decided to go one step further. In our 2014 Spring Release we announced native support for MongoDB in our Database Monitoring product. This release provides stand alone, enterprise-class tooling for MongoDB and highlights just how creaky and tired the monitoring supplied by the legacy database vendors has become. So, now application owners, DevOps teams, and DBAs can have monitoring for MongoDB whether as part of an end-to-end monitoring platform or as a stand-alone deployment for inspecting the MongoDB database. Identifying root causes of application latency is only a couple clicks away – without a bunch of training, a long installation process, or any of the baggage that came with the old generation of enterprise IT systems.

We recently caught up with Matt Asay, VP of Marketing and Business Development at MongoDB, to discuss the partnership. Check out the video below to see how customers are using AppDynamics and MongoDB as critical components in the new generation IT shop.

One of the reasons we wanted to work with AppDynamics is to give our users, our end customers, that holistic view the deep view of what’s going on in their application … We would like our customers to get to that root cause of the problem sooner rather than later … It’s fantastic working with AppDynamics.

Want to learn more about the future of Enterprise IT and NoSQL? Be sure to register for MongoDB World 2014 taking place in New York, June 24 & 25.  You’ll learn everything you need to know to build and manage modern applications built on MongoDB. AppDynamics will be in booth #111.

Want to start monitoring your MongoDB database? Check out a free trial of AppDynamics today!

Announcing AppDynamics Application Intelligence Platform

AppDynamics is excited to unveil details behind our new Application Intelligence Platform which is the technology foundation for the AppDynamics portfolio of performance monitoring, automation and analytics solutions (transaction analytics was also announced today).

Today’s software defined businesses need the ability to see everything in their application environment, act with certainty, and know faster. See, Act, & Know, the three core pillars of Application Intelligence.

See Act Know

Making sense of all the data, transactions, events and user interactions in these environments is increasingly difficult as the volume, variety and velocity of data grows. Traditional approaches to solving this problem are typically silo’d in their view providing limited business correlation and only present static data. In order to keep up with todays complexity, what’s needed is a platform that can collect, understand, and act on the data and translate that into actionable information about your business.

platform page - application intelligence platform graphic2

The AppDynamics Application Intelligence Platform is uniquely able to deliver rich performance data, learning, and analytics, combined with the flexibility to adapt to virtually any infrastructure or software environment. The platform was designed for the modern day application, including SOA architectures, frequent agile releases, cloud deployments, big data technologies and mobile delivery – and maintains a unique low-overhead architecture. This gives you the ability to instantly see the impact of production changes as you work, and adapt to optimize performance and minimize bottlenecks while ensuring the revenues are maximized.

Today’s consumer-driven world moves faster than ever. Everything from your customer experience to revenue depends on your business-critical applications performing at their highest level. AppDynamics delivers real-time access to every aspect of your performance, so you can anticipate problems, resolve them automatically, and make smarter, more certain business decisions.  This includes detailed understanding of the user experience, business data such as real time revenue metrics, deep application performance and infrastructure data. All collected within context of business & operational transactions.

Platform Design Principles:

The Application Intelligence Platform is the underlying architecture that we use to deliver all of the capabilities a modern business needs to keep their business running at peak performance and use application intelligence as a competitive advantage.

The core design principles behind the AppDynamics Application Intelligence Platform include:


The interactions customers have with the business are highly dynamic, environments are larger and more hybrid and organizations are demanding IT transform the business faster than ever. AppDynamics monitors many of the largest, most complex enterprise application environments in the world, supporting environments up to 10,000 nodes on a single management server. This is critical for customers who do not want to manage and maintain a farm of servers.


Our platform was designed to be a production tool from the very beginning, giving it a unique low-overhead architecture. Intelligent data collection to ensure the lowest production overhead in the market. AppDynamics correlates end user business transaction details with completion status (success, error, exception), response times, and all other data points measured at any given time. It automatically analyzes the entire data set to provide information from which to draw conclusions and take the appropriate action.


Auto-instrumentation, and minimal configuration enables simple and intuitive operation for any size business. AppDynamics has a unique ability to instrument your application, learning its transactions, code execution behavior, and normal performance patterns. It can adapt and adjust instrumentation automatically when you change your application-no manual configuration needed.

Open and Extensible:

The AppDynamics platform is open, extensible, and interoperable to fit any business need. Through the AppDynamics Exchange our community comes together to share knowledge and contribute back over 100 extensions that provide deep integrations to the tools you already use like Splunk, Apica, PagerDuty, and Amazon Web Services. From monitoring Amazon Web Services costs to MEAS mainframes our extensions allow you to leverage the tools you already have in place. We believe you own your data and it should be easy to consume and analyze which is why we provide REST APIs and SDKs available on GitHub that make it as easy as possible to get started.


The platform can be deployed in any operating model, including on-premise, SaaS, private cloud, or a hybrid combination. It’s always the exact same platform and there are no cost implications associated regardless of deployment model.


Provides enterprise grade security with Role-Based Access Control (RBAC), LDAP, and SaaS certifications that guarantee security and compliance.

What services are delivered on the Application Intelligence Platform?

The AppDynamics Application Intelligence Platform gives companies the foundation they need to monitor performance and extract meaningful analytics from their business applications. Software-defined businesses must be certain that their most complex, business-critical applications are performing at the highest level; and be certain that the data and information generated by these applications can be harnessed for ongoing business advantage and impact. AppDynamics’ Application Intelligence platform enables customers to monitor, respond, and analyze their application environment.


Application Performance Management

Mobile Application Monitoring

End User Experience Management

Database/NoSQL/BigData Monitoring

Infrastructure Monitoring


Run Book Automation

Cloud Auto-Scaling



Transaction Analytics (Beta)


Real-Time Business Metrics

analytics page - real-time business metrics graphic2

Operational Analytics

scalability analysis

The Application Intelligence Platform delivers this functionality by collecting data from across the application environment, processing & correlating it, and converting that data into knowledge for customers.

Collecting Data:

  • Instrumentation allows us to watch every line of code.
  • Distributed transaction tracing follows the business transaction across all tiers of the application, not just limited to a single node.

  • Transaction auto-learning engine inspects execution code, payload, libraries and methods so you are not left in the dark.

  • Real-time service discovery renders architecture topology to clearly and automatically map critical relationships and dependencies..

  • The smart agent filters and transfers data to the management server securely and in real-time.

Processing & Correlating Data

  • Real-time stream processing allows for complex event processing of metric and event streams.
  • Time series clustering and analytics provides the ability to index and manage time series by auto rolling up, purging or clustering data sets by time increments.

  • Behavioral learning is an engine that continuously adjusts dynamic baselines for an automated manageable data set.

  • Unstructured and structured big data indexing creates a data warehouse for different types of application data which is seamlessly correlated and processed.

  • Event correlation shows the relationship of performance, change, and business events.

Converting Data Into Knowledge

  • Intuitive user interface (UI) allows for a simple and easy-to-use platform for business users, developers and operations teams.

  • Dynamic flow maps create a clear and understandable visual representation of the application topology, so organizations understand what is happening in the big picture.

  • Cross-correlated drill down from anywhere in the UI, enabling pin-point accuracy across multiple nodes and machines.

  • Real-time correlation of performance metrics and business metrics create a business understanding of data never before delivered.

  • Compare and analyze the side-by side performance of different versions of an application.

  • Custom drag and drop HTML5 dashboards can be configured to any specification to deliver clear actionable results.

  • Query language allows easy, targeted data discovery.

In today’s competitive and demanding environment, organizations need to leverage Application Intelligence for ongoing business advantage. The AppDynamics Application Intelligence Platform enables customers to see everything across all front and back-end interactions: business transaction and code-level visibility with no blind spots.

Maximize digital business performance through an open, integrated solution that delivers real-time monitoring, actionable business and operational insights, and automated issue resolution across applications, infrastructure, and user experience. Take five minutes to get complete visibility and control into the applications that power your software-defined business. Get started with the AppDynamics Application Intelligence Platform today.

Database Monitoring for MariaDB and Percona Server

Both MariaDB and Percona Server are forks of MySQL and strive to be drop in replacements for MySQL from a binary, api compatibility, and command line perspective.

It’s great to have an alternative to MySQL since you never know what might happen to it given that Oracle bought it for 1 billion dollars. In this blog post I set out to see if these MySQL forks would work 100% with AppDynamics for Databases. If you’re not familiar with the AppDynamics for Databases product I suggest you take a few minutes to read this other blog post.

The Setup

Getting both MariaDB and Percona Server installed onto test instances was pretty simple. I chose to use 2 Red Hat Enterprise Linux (RHEL) servers running on Amazon Web Services (AWS) for no particular reason other than they were quick and easy to get running. My first step was to make sure that MySQL was gone from my RHEL servers by running “yum remove mysql-server”.

Installing both MariaDB and Percona Server consisted of setting up yum repository files (documented here and here) and running the yum installation commands. This took care of getting the binaries installed so the rest of the process was related to starting and configuring the individual database servers.

The startup command for both MariaDB and Percona Server is “/etc/init.d/mysql start” so you can see that these products really do strive for direct drop in adherence to MySQL. As you can see in the screen grabs below I ended up running MariaDB 10.0.3 and Percona Server 5.5.31-30.3.

Screen Shot 2013-07-01 at 2.21.48 PM Screen Shot 2013-07-01 at 2.23.30 PM

Connected to each of these databases were 1 instance of WordPress and 1 instance of Drupal in a nearly “out of the box” configuration besides adding a couple of new posts to each CMS to help drive a small amount of load. I didn’t want to set up a load testing tool so I induced a high disk I/O load on each server by running the UNIX command “cat /dev/zero > /tmp/zerofile”. This command pumps the number 0 into that file as fast as it can basically crushing the disk. (Use Ctrl-C to kill this command before you fill up your disk.)

The Monitoring

Getting the monitoring set up was really easy. I used a test instance of AppDynamics for Databases to remotely monitor each database instance (yep, no agent install required). To initiate monitoring I opened up my AppDynamics for Databases console, navigated to the agent manager, clicked the “add agent” button, and filled in the fields as shown below (I selected MySQL as the database type):

Screen Shot 2013-07-01 at 3.39.47 PM

My remote agent didn’t connect the first time I tired this because I forgot to configure iptables to let my connection through even though I had set up my AWS firewall rules properly (facepalm). After getting iptables out of the way (I just turned it off since these were test instances) my database monitoring connections came to life and I was off and running.

The Result

Taking a look at all of the data pouring into AppDynamics for Databases I can see that it is 100% compatible with MariaDB and Percona Server. There are no errors being thrown and the data is everything that it should be.

The beauty of my induced disk I/O load was that just by clicking around the web interface of WordPress and Drupal I was getting slow response times. That always makes data more interesting to look at. So here are some screen grabs for each database type for you to check out…

MariaDB Activity

AppDynamics for Databases activity screen for the MariaDB database.

Percona Activity

AppDynamics for Databases activity screen for the Percona database.

MariaDB Explain Statement

Explain statement for a select statement in MariaDB.

Percona Explain Statement

Explain statement for a select statement in Percona.

MariaDB Statistics

A couple of statistics charts for MariaDB.

Percona Statistics

A couple of statistics charts for Percona.

If you’re currently running MySQL you might want to check out MariaDB and Percona Server. It’s possible that you might see some performance improvements since the storage engine for MariaDB and Percona is XtraDB as opposed to MySQL’s InnoDB. Having choices in technology is a great thing. Having a unified monitoring platform for your MySQL, MariaDB, Percona Server, Oracle, SQL Server, Sybase, IBM DB2, and PostgreSQL database is even better. Click here to get started with your free trial of AppDynamics for Databases today.

How To Set Up and Monitor Amazon RDS Databases

Relational databases are still an important application component even in today’s modern application architectures. There is usually at least one relational database lurking somewhere within the overall application flow and understanding the behavior of these databases is major factor in rapidly troubleshooting application problems. In 2009, Amazon launched their RDS service which basically allows anyone to spin up a MySQL, Oracle, or MS-SQL instance whenever the urge strikes.

While this service is amazingly useful there are also some drawbacks:

  1. You cannot login and access the underlying OS of your database instance. This means that you can’t use any agent based monitoring tools to get the visibility you really want.
  2. The provided CloudWatch monitoring metrics are high level statistics and not helpful in troubleshooting SQL issues.

The good news is that you can monitor all of your Amazon RDS instances using AppDynamics for Databases (AppD4DB) and in this article I will show you how. If you’re unfamiliar with AppD4DB click here for an introduction.

Setting Up A Database Instance In RDS

Creating a new database instance in RDS is really simple.

Step 1, login to your Amazon AWS account and open the RDS interface.

 RDS 1

Step 2, Initiate the “Launch a DB Instance” workflow.


Step 3, select the type of instance you want to launch. In this case we will use MySQL but I did test Oracle and MS-SQL too.


Step 4, fill in the appropriate instance details. Pay attention to the master user name and password as we will use those later when we create our monitoring configuration (although we could create a new user only for monitoring if we want).


Step 5, finish the RDS workflow. Notice I called the database “wordpress” as I will use it to host a WordPress instance. Also notice that we chose to use the “default” DB security group. You will need to access the security group settings after your new instance is created so that you allow access to the database from the internet. For the sake of testing I opened up my database to (not shown in this workflow) which allows the entire internet to connect to my database if they have the credentials. You should be much more selective if you have a real database instance with production applications connected.




Step 6, wait for your instance to be created and watch for the “available” status. When you click on the database instance row you will see the details populate in the “Description” tab below. We will use the “Endpoint” information to connect AppD4DB to our new instance. (At this point you can also build the database structure and connect your application to your running instance.)


Monitor Your Database With AppD4DB

Step 1, enable database monitoring from the “Agent Manager” tab in AppD4DB. Notice we map RDS “Endpoint” to AppD4DB “Hostname or IP Address” and in this case we are using the RDS “Master Username” and “Master Password” for “Username” and “Password” in AppD4DB. Also, since Amazon does not allow any access to the associated OS (via SSH or any other method) we cannot enable OS monitoring.


Step 2, start your new database monitoring and use your application. Here is a screen grab showing a couple of slow SQL queries.

RDS SQL Activity

The Results

So here is what I found for each type of database offered by Amazon RDS.

  • MySQL: Fully functional database monitoring.
  • Oracle: Fully functional database monitoring.
  • MS-SQL: All database monitoring functionality works except for File I/O Statistics. This means that we are 99% functional and capture everything else as expected including the ability to show SQL execution plans.

Amazon RDS makes it fast and easy to stand up MySQL, MS-SQL and Oracle databases. AppDynamics for Databases makes it fast and easy to monitor your RDS databases at the level required to solve your application and database problems. Sounds like a perfect match to me. Sign up for your free trial of AppD4DB and see for yourself today.

Introducing AppDynamics for Databases

The most common cause of application slowdowns today is slow SQL or stored procedures in the database. The reason? Databases store large amounts of data on disk, and disk is super slow relative to data in memory (unless its SSD). As data volumes grow, the need to configure, maintain and optimize a database also grows because you can’t manage everything on one disk or volume. This is why database administrators (DBA’s) exist.

A huge problem is that developers write ad-hoc SQL queries for their applications and have no idea how those queries actually execute and retrieve data from disk in the database. They are blind to the number of records in a table, or which columns in a given table has indexes. This is why developers typically blame the DBA when their SQL queries run slow– they assume slow queries means a slow database, and that the DBA isn’t doing his job properly. This couldn’t be further from the truth.

With AppDynamics for databases we’re now giving DevOps teams visibility of how their application SQL queries actually execute within the database, so when things slow down they can now collaborate with DBA’s instead of blaming them. Thats right, users of AppDynamics will now be able to find the root cause of slow SQL and stored procedures in their application. This is indeed good news because customers can dramatically improve their end user experience, response time and application throughput by tuning their data access layer.

Obviously, SQL executes differently in different databases, thats why we built AppDynamics for databases to provide universal support for all common relational databases like Oracle, SQL Server, DB2, Sybase, MySQL and PostGres. This is a big deal because no monitoring vendor at the moment provides database diagnostics across all type of relational databases, not to mention providing the application and business transaction context.

So let’s show you a few screenshots of what AppDynamics for Databases can give you.

Single Pane of Glass View
How about a single pane of glass view to monitor ALL your databases regardless of platform? done. Application databases these days vary significantly, most use Oracle, some use MySQL and obviously .NET applications are better related to windows-based databases like SQL Server and Sybase. Its therefore important you can get a holistic view of performance across all your databases.

Real-Time Performance Metrics:
Its always good to know exactly what is happening “right now” in a database. Has the database ran out of connections? or is the database experiencing latency from locking on rows? These are just a few answers you can get from the below “Current” workspace which provides a real-time view of database resources and performance.

Screen Shot 2013-03-26 at 4.52.40 AM

Historical Analysis of Database Activity:
Real-time data is good, but its also helpful to have historical data so you can identify trends, spikes and abnormal patterns in performance. For example, a simple application code change can have a dramatic impact on a database and its performance.  We had an agile customer last year who deployed a code release in production and immediately saw a slowdown in application response time. When they drilled into AppDynamics for Java they noticed that one business transaction was now performing 25 SQL queries per execution instead of just 2 queries, this was the difference between 20,000 and 250,000 executions per minute in the database. Obviously when you increase concurrency that much in the database your going to experience contention and wait, being able to visually track database resource, time spent, wait states and number of executions over-time is invaluable.

Screen Shot 2013-03-26 at 4.53.00 AM

Top SQL Statements and Stored Procedures:
Perhaps the most obvious view you’d expect from a database monitoring solution. This is typically what most Application Performance Monitoring (APM) vendors provide by extracting data from the JDBC or ADO.NET protocols. The big difference between those solutions and AppDynamics is that we allow you to drill down into the SQL or stored procedures and understand their execution plans, so you can actually find the root cause of why your queries run slow. This is great data for application support teams and developers who want to collaborate better with their DBA’s, so they can understand the real reason of database latency.

Screen Shot 2013-03-26 at 4.53.09 AM

Explain Plans:
Most databases will automatically parse and refine the execution of SQL queries based on what plan its query optimizer selects. Just because you add an index to a table to try and make a query faster, doesn’t mean the database query optimizer will use it, and when you consider that index’s aren’t for free (they take up disk space) you might want to check exactly how the database is executing your queries. Take a look at the below explain plan and you can see how the SQL is being processed with two simple selects and two different tables. Notice how the SUBQUERY on the iplookup table is using the index ‘ian1’ because that table has over a million rows in it. If this index was accidentally dropped you can be sure this query would run significantly slower given it would be doing a full table scan on over a million records.

Screen Shot 2013-03-26 at 4.53.17 AM

Database Statistics:
How frequently are applications connecting to your database? and what operations are these applications performing? Applications differ by the volume of data they request, process and manage over-time. For example, in a Cable or Telco provider you might have a customer portal application which accesses customer data inside a large Oracle schema. That same Oracle schema may also service queries from reporting applications for marketing, or perhaps batch jobs from billing applications that need to process large volumes of customer data. If you have different applications performing different operations (read, write, update, delete) at the same time on the same database then that can be a recipe for disaster. You can see from the below screenshot that the application connect to this MySQL instance is spending most of its time inserting data, meaning its write intensive versus read intensive. Obviously if the database and application was purely read intensive, you might consider moving that data to a cache in memory closer to the application logic. Remember, database calls are often remote and expensive, ever so more in the cloud where storage is less than stella.

Screen Shot 2013-03-26 at 4.53.25 AM

 Object Properties:
All database schema’s have generic objects that represent users, databases, tables, indexes and so on. These objects are often configured and maintained by DBA’s to ensure that the database is optimally configured for availability and performance. Change is constant within databases because data volumes are always increasing, and application data models are always evolving to support new features. Making a single configuration change can have a dramatic impact on database and application performance. AppDynamics for databases flags and provides a full audit report on all changes made within the database as shown below. This helps DevOps and DBA’s correlate the impact of change with database performance which can be very powerful. If someone dropped an index and performance spikes shortly after then thats worth knowing!

Screen Shot 2013-03-26 at 4.53.35 AM

 I walked through just a few features and capabilities of AppDynamics for Databases, here’s a quick 4 minute overview of the product:

You can get started right now and sign up for our free trial.

Happy database monitoring!


Data Clouds Part II: My Big Data Dashboard

In my previous blog, I wrote at length about the complexities of running a data cloud in production. This logical data set, spread across many nodes, requires a whole new set of tools and methodologies to run and maintain. Today we’ll look at one of the biggest challenges in managing a data cloud – monitoring.

Database monitoring used to be easy in the days before data clouds. Datasets were stored in a single large database, and there were hundreds of off-the-shelf products available to monitor the performance of that database. When problems occurred, one had simply to open up the monitoring tool and look at a set of graphs and metrics to diagnose the problem.

There are no off-the-shelf tools for monitoring a data cloud, however. There’s no easy way to get a comprehensive view of your entire data cloud, let alone diagnose problems and monitor performance. Database monitoring solutions simply don’t cut it in this kind of environment. So how do we monitor the performance of our data cloud? I’ll tell you what I did.

It just so happens I work at AppDynamics, one of the most powerful application monitoring tools on the market. We monitor all parts of your application including the data layer, with visibility into both Relational and NoSQL systems like Cassandra. With AppDynamics I was able to create a dashboard that gives me a single pane-of-glass view into the performance of my data cloud.

Big Data Dashboard

My Big Data Dashboard

This dashboard is now used in several departments at AppDynamics including Operations, QA, Performance and development teams to see how our data cloud is running. All key metrics about all of our replicas are graphed side by side on one screen. This is the dream of anyone running big data systems in production!

Of course, not all problems are system wide. More often than not you need to drill into one replica or replica set to find a problem. To do that, I simply double click on any part of my big data dashboard to focus on a single replica, change the time range, and add more metrics.

Data clouds are difficult to run, and there aren’t any database monitoring tools fit to monitor them yet. But instead of sitting around waiting for data monitoring tools to catch up with our needs, I’ve built my own Big Data Dashboard with monitoring tool designed for applications.

Of course the fun doesn’t stop here…I still need to find a way to set up alerts and do performance tuning for my data cloud. Stay tuned for more blogs in this series to see how I do it!