This summer, I had a chance to sit on a panel at DevOps Days with a few industry colleagues to discuss the role of monitoring, testing, and performance in solving DevOps issues. It was a lively discussion on the teams, tools, and techniques that play a role in managing application and network performance.

Read the Full Post…

APM Thought Leadership, application development, DevOps, IT operations

It’s a big day for us here at AppDynamics as we mark (and pass!) the 5000th download of AppDynamics Lite! Released just a few months ago, AppDynamics Lite is our free APM tool for troubleshooting Java performance in production.  We hear every day from companies who have deployed Lite in production environments and are using it to solve real problems.  That’s why we built it, and it’s very rewarding to hear about IT Operations professionals and developers getting real value.

This conversation represents another goal of ours: removing the friction that typically exists between software companies and evaluators. With Lite, companies can download it whenever they want and put it immediately into action.  No waiting, no negotiating…just instant access.  This strategy works for us because, as a startup, we believe that if someone becomes familiar with our solution, they’ll see that it’s considerably better than the legacy offerings on the market.

APM Thought Leadership, appdynamics, appdynamics lite, application monitoring, cloud computing, developers, DevOps

This post is part of our series on the Top Application Performance Challenges.

Over the past couple of weeks we helped a few DevOps groups troubleshoot performance problems in their production Java-based applications. While the applications themselves bore little resemblance to one another, they all shared one common problem. They were full of runtime exceptions.

When I brought the exceptions to the attention of some of the teams I got a variety of different answers.

• “Oh yeah, we know all about that one, just some old debug info … I think”
• “Can you just tell the APM tool to just ignore that exception because it happens way too much”
• “I have no idea where that exception comes from, I think it’s the vendor/contractor/off shore team’s/someone else’s code”
• “Sometimes that exception indicates an error, sometimes we use it for flow control”

The response was the same – because the exception did not affect the functionality of (read: did not break) the application it was deemed unimportant. But what about performance?

In one high-volume production application (100’s of calls/second) we observed an NPE (Null Pointer Exception) rate of more than 1000 exceptions per minute (17,800/15 min). This particular NPE occurred in the exact same line of code and propagated some 20 calls up in the stack before a catch clause handled it. The pseudo code for the line read if the string equaled null and the length of the trimmed string was zero then do x otherwise just exit. When we read the line out loud the problem was immediately obvious – you cannot trim a null string. While the team agreed that they needed to fix the code, they remained skeptical that simply changing ‘=’ to ‘!’ would really improve performance.

Using basic simulations we estimate this error rate imposes a 3-4% slow down on basic response time at a minimum. You heard me, 3-4% slow down at a minimum.

And these response issues get compounded in multi-threaded application server environments. When multiple pooled threads slow down to cope with error handling then the application has fewer resources available to process new requests. At the JVM level the errors cause more IO, more objects created in the heap, more garbage collection. In a multi-threaded environment the error threatens more than response times, it threatens load capacity.

And if the application is distributed across multiple JVM’s … well you get the picture.

The web contains many articles and discussions on the performance impact of Java’s exception handling. The examples in these posts provide sufficient data to show that a real performance penalty exists for throwing and catching exceptions. Doing the fix really does improve the performance!

Runtime Exceptions happen. When they occur frequently, they do appreciably slow down your application. The slowness becomes contagious to all transactions being served by the application. Don’t mute them. Don’t ignore them. Don’t dismiss them. Don’t convince yourself they are harmless. If you want a simple way to improve your applications performance, start by fixing up these regular occurring exceptions. Who knows, you just might help everyone’s code run a little faster.

APM Thought Leadership, application, application monitoring, developers, DevOps, IT operations, runtime exception, slow response error

Interested in winning an iPad? Take our Application Virtualization survey, and we’ll give you a more-than-decent shot at winning your very own!

In talking to our customers, AppDynamics recognizes that virtualizing mission-critical applications is at the top of everyone’s mind, even though some companies are at different stages of the process than others.

Many IT departments have already finished virtualizing their less critical apps and they realize that there are many benefits to be had in virtualizing the rest. But, in many cases, application owners are concerned about performance impacts to their mission-critical apps, and they’re saying, “Hands off my app.”

What’s it like at your organization? Is the path to app virtualization a smooth one, or are you experiencing some bumps along the way? We’d like to find out more, and we’d like your help to do it.

Answer a few short questions on your app virtualization strategy and be automatically entered to win an iPad. In addition, we’ll send you a copy of the results, so you can discover what your peers at other organizations are doing.

Find out:

- What percentage of Tier 1 apps have been virtualized by other companies
- Whether the virtualization project is a stepping stone towards the cloud
- What challenges are preventing people from finishing (or even starting) virtualization of Tier 1 apps

The survey is only 10 questions long–and again, entering gives you the possibility of winning an iPad, as well as a guaranteed copy of the survey results.

Take the survey now!

We look forward to reviewing your responses!

APM Thought Leadership, application, application development, cloud computing, virtualization

This is the first post in a series on the Top Application Performance Challenges.

Of the many issues affecting the performance of Java/.NET applications, synchronization ranks near the top.  Issues arising from synchronization are often hard to recognize and their impact on performance can be become significant. What’s more, they are often, at least in principle, avoidable.

The fundamental need to synchronize lies with Java’s support for concurrency. This is implemented by allowing the execution of code by separate threads within the same process. Separate threads can share the same resources, objects in memory. While being a very efficient way to get more work done (while one thread waits for an IO operation to complete, another thread gets the CPU to run a computation), the application is also exposed to interference and consistency problems.

The JVM/CLR does not guarantee an execution order of the code running in concurrent threads. If multiple threads reference the same object there is no telling what state that object will be in at a given moment in time. The repercussions of that simple fact can be enormous with, for example, one thread running calculations and returning wrong results because a concurrent thread accessing and modifying shared bits of information at the same time.

To prevent such a scenario (a program needs to execute correctly, after all) a programmer uses the “synchronize” keyword in his/her program to force order on concurrent thread execution. Using “synchronize” prevents threads from obtaining the same object at the same time.

In practice, however, this simple mechanism comes with substantial side effects. Modern business applications are typically highly multi-threaded. Many threads execute concurrently, and consequently “contend” heavily for shared objects. Contention occurs when a thread wants to access a synchronized object that is already held by another thread. All threads contending effectively “block,” halting their execution until they can acquire the object. Synchronization effectively forces concurrent processing back into sequential execution.

With just a few metrics we can show the effects of synchronization on an application’s performance. For instance, take a look at the graph below.

While increasing load (number of users = blue), we see that at some point midway the response time (yellow) takes an upward curve, while at the same time resource usage (CPU = red) somewhat increases to eventually plateau and even recedes. It almost looks like the application runs with the “handbrake on,” a classic, albeit high-level, symptom of an application that has been “over-synchronized.”

With every new version of the JVM/CLR improvements are made to mitigate this issue. However, while helpful, these improvements can’t fully resolve the issue and address the application’s negative performance.

Also, developers have come to adopt “defensive” coding practices, synchronizing large pieces of code to prevent possible problems. In large development organizations this problem is further magnified as no one developer or team has full ownership of an application’s entire code base. The practice to err on the side of safety can quickly exacerbate with large portions of synchronized code significantly impacting the performance of an application’s potential throughput.

It is often too arduous a task to maintain a locking strategy fine grained enough to ensure that only the necessary minimum of execution paths are synchronized. New approaches to better manage state in a concurrent environment are available in newer versions of Java such as readWriteLocks, but they are not widely adopted yet.  These approaches promise a higher degree of concurrency, but it will always be up to the developer to implement and use the mechanism correctly.

Is synchronization, then, always going to result in a high MTTR?

New technologies exist on the horizon that may lend some relief.  Software Transactional Memory Systems (STM), for example, might become a powerful weapon for dealing with synchronization issues. They may not be ready for prime time yet, but given what we’ve seen with database systems, they might be the key to taming the concurrency challenges affecting applications today. Check out JVSTM, Multiverse and Clojure for examples of STMs.

For now, the best development organizations are the ones that can walk the fine line of balancing code review/rewrite burdens and concessions to performance. APM tools can help quite a lot in such scenarios, allowing to monitor application execution under high load (aka “in production”) and quickly pinpoint to the execution times for particular highly contended Objects, Database connections being a prime example. With the right APM in place, the ability to identify thread synchronization issues become greatly increased—and the overall MTTR will drop dramatically.

APM Thought Leadership, application development, developers, synchronization, thread

I’ve never written a blog post before, but Jyoti tells me, from experience, that if you write about your passion, it’s a piece of cake. So here goes. My passion is to create technology to make application management easier and more productive so that companies can focus on delivering value instead of fretting about just being able to run apps 24×7. Since I intend to keep up this newly acquired habit of mine, let me start by establishing the fundamental tenet on which this technology we build at AppDynamics is based – Business Transactions.

You may think you’ve heard this discussion before, but I’d wager a guess that you’ve actually heard more about transactions than about business. I want to tell you why you actually need to put the focus on the business.

In today’s world where more and more businesses move to the web, the application is the face of the company. It is the business, it is the revenue stream. From DVD rentals to talent management, everything is an online application. Someone suggested to me recently that the only business they cannot use the internet for was getting a haircut. But I am sure they are working on it too!!

From the CIO to the ops team pushing the latest application release out, there needs to be a common context that drives the business unit towards a robust, efficient and a highly competitive application. That context is the business transaction.

Here are a few reasons why you should focus on the ‘business’ side of the business transaction:

1. When the application is the business, competitive advantage means having a better application. Now in an ideal world, when someone builds an application they would like it to be better than anything out there. But having an edge is not a one time thing. It’s constant evolution, which means rapid change and faster adaptation (Flickr does 10+ deployments a day!!). Dynamic changes to an application’s features and functions directly impact the business transaction and the overall user experience. At the end of the day, what you care about most is how your users are being serviced.

2. When the application is the business, managing an application is no longer just about monitoring CPU, JVM memory or timing key methods. It’s really about understanding the user experience, managing the business operations and creating service levels to ensure optimum performance. Business transactions are the binding factor for attaching SLAs to business operations and for creating a common ground between dev teams and ops teams. Focusing on the business transaction is a great way to validate the whole cycle including a rollout or the current state of the app at any given time.  Also, the growth of business means more capacity to serve. This means the growth of servers. So, instead of multiplying management complexity with more resources, a focus on user-centric SLAs is the only scalable way to address performance.

3. When the application is the business, the responsibility of managing applications (and hence the revenue stream) falls on IT Operations. But development is responsible for building the app and for the innovation in it. The creates an interdependence (and what we see as the DevOps movement). By creating a common language to categorize user requests – “Are the checkouts doing ok?” “Is the sales order processing running faster than before?” – these teams are better able to communicate and effectively manage application performance over time.

Using the business transaction as a unit of management for the application makes your company more agile, competitive, scalable and high performance. In future posts, I’ll review the various ways to approach overall application management and take a deeper dive on why a business transaction focus is critical to success.

At the end of the day, it should be all about the business!

APM Thought Leadership, appdynamics, application, application monitoring, DevOps