Taking a break from the rush of activity at VMWorld, today we released the results of our first Application Virtualization Outlook survey.  With virtualization a top priority for many CIOs in the next year, we wanted to hear from IT professionals about their plans to virtualize mission-critical apps. Most importantly, we wanted to learn how quickly they’re making the transition, or whether there are obstacles getting in their way.

We found the survey results pretty interesting – overall, it seems like many application owners recognize the potential benefits of virtualization, but lack the confidence that their Tier-1 apps will continue to perform as needed when moved to a virtual environment. Let’s take a closer look at a few of these results and how we came to this conclusion:

More than 80 percent of professionals polled had virtualized their non-critical systems, but on the flip side, only 14 percent had virtualized more than three quarters or more of their Tier 1 apps (see charts below).  That’s a big divide in the pace of adoption.

The number one obstacle to virtualizing apps? “People Issues,” i.e., application owners who impede the virtualization process.  Respondents also cited worries about performance degradation and application design.

And one of our favorite questions- almost one-third of respondents reported that there are people in their organization who would say, “My Tier 1 application will be virtualized over my dead body!!”  This definitely proves that application virtualization is can elicit an emotional response in concerned application owners.

But, let’s not get discouraged. Just about everyone surveyed agreed that there are some great benefits to be had with virtualization. In particular, respondents noted that server consolidation, power and cost savings were likely payoffs, along with disaster recovery and agility improvements.

So how can virtualization teams assuage the concerns of their colleagues and reap these benefits? By providing hard evidence that the owners’ application will perform well in a virtual environment.

Application owners are responsible for meeting SLAs and for maintaining 100% up-time – and they’re not going to hand over their application without proof that those business objectives won’t be compromised.

The best way to get these facts is to conduct tests to establish a baseline performance in a non-virtual environment, and then provide a comparison of application performance, pre-and post-virtualization.  This “apples-to-apples” strategy presents app owners with evidence that their application will continue to perform to their high standards in a virtual environment.

For more information about how we help companies manage application performance in virtual environments, check this out.

application monitoring, application performance management, cloud computing, virtualization

This summer, I had a chance to sit on a panel at DevOps Days with a few industry colleagues to discuss the role of monitoring, testing, and performance in solving DevOps issues. It was a lively discussion on the teams, tools, and techniques that play a role in managing application and network performance.

We all agreed that visibility is critical – but it’s not a lack of data that can challenge DevOps teams; it’s the knowledge of that data and the ability to get meaningful information out of it that is key to success. It’s also about information sharing.

Like we say all the time – creating a common context for understanding monitoring metrics and performance goals is hugely important. The best way to do this is to identify the business goals and business metrics early on and build those into your monitoring metrics. That way both teams are working towards the same goals and sharing responsibility for success.

To hear the full panel discussion from DevOps Days, check out the video on InfoQ here.

application development, application performance management, DevOps, IT operations

It’s a big day for us here at AppDynamics as we mark (and pass!) the 5000th download of AppDynamics Lite! Released just a few months ago, AppDynamics Lite is our free APM tool for troubleshooting Java performance in production.  We hear every day from companies who have deployed Lite in production environments and are using it to solve real problems.  That’s why we built it, and it’s very rewarding to hear about IT Operations professionals and developers getting real value.

This conversation represents another goal of ours: removing the friction that typically exists between software companies and evaluators. With Lite, companies can download it whenever they want and put it immediately into action.  No waiting, no negotiating…just instant access.  This strategy works for us because, as a startup, we believe that if someone becomes familiar with our solution, they’ll see that it’s considerably better than the legacy offerings on the market.

appdynamics, appdynamics lite, application monitoring, application performance management, cloud computing, developers, DevOps

This post is part of our series on the Top Application Performance Challenges.

Over the past couple of weeks we helped a few DevOps groups troubleshoot performance problems in their production Java-based applications. While the applications themselves bore little resemblance to one another, they all shared one common problem. They were full of runtime exceptions.

When I brought the exceptions to the attention of some of the teams I got a variety of different answers.

• “Oh yeah, we know all about that one, just some old debug info … I think”
• “Can you just tell the APM tool to just ignore that exception because it happens way too much”
• “I have no idea where that exception comes from, I think it’s the vendor/contractor/off shore team’s/someone else’s code”
• “Sometimes that exception indicates an error, sometimes we use it for flow control”

The response was the same – because the exception did not affect the functionality of (read: did not break) the application it was deemed unimportant. But what about performance?

In one high-volume production application (100’s of calls/second) we observed an NPE (Null Pointer Exception) rate of more than 1000 exceptions per minute (17,800/15 min). This particular NPE occurred in the exact same line of code and propagated some 20 calls up in the stack before a catch clause handled it. The pseudo code for the line read if the string equaled null and the length of the trimmed string was zero then do x otherwise just exit. When we read the line out loud the problem was immediately obvious – you cannot trim a null string. While the team agreed that they needed to fix the code, they remained skeptical that simply changing ‘=’ to ‘!’ would really improve performance.

Using basic simulations we estimate this error rate imposes a 3-4% slow down on basic response time at a minimum. You heard me, 3-4% slow down at a minimum.

And these response issues get compounded in multi-threaded application server environments. When multiple pooled threads slow down to cope with error handling then the application has fewer resources available to process new requests. At the JVM level the errors cause more IO, more objects created in the heap, more garbage collection. In a multi-threaded environment the error threatens more than response times, it threatens load capacity.

And if the application is distributed across multiple JVM’s … well you get the picture.

The web contains many articles and discussions on the performance impact of Java’s exception handling. The examples in these posts provide sufficient data to show that a real performance penalty exists for throwing and catching exceptions. Doing the fix really does improve the performance!

Runtime Exceptions happen. When they occur frequently, they do appreciably slow down your application. The slowness becomes contagious to all transactions being served by the application. Don’t mute them. Don’t ignore them. Don’t dismiss them. Don’t convince yourself they are harmless. If you want a simple way to improve your applications performance, start by fixing up these regular occurring exceptions. Who knows, you just might help everyone’s code run a little faster.

application, application monitoring, application performance management, developers, DevOps, IT operations, runtime exception, slow response error

Interested in winning an iPad? Take our Application Virtualization survey, and we’ll give you a more-than-decent shot at winning your very own!

In talking to our customers, AppDynamics recognizes that virtualizing mission-critical applications is at the top of everyone’s mind, even though some companies are at different stages of the process than others.

Many IT departments have already finished virtualizing their less critical apps and they realize that there are many benefits to be had in virtualizing the rest. But, in many cases, application owners are concerned about performance impacts to their mission-critical apps, and they’re saying, “Hands off my app.”

What’s it like at your organization? Is the path to app virtualization a smooth one, or are you experiencing some bumps along the way? We’d like to find out more, and we’d like your help to do it.

Answer a few short questions on your app virtualization strategy and be automatically entered to win an iPad. In addition, we’ll send you a copy of the results, so you can discover what your peers at other organizations are doing.

Find out:

- What percentage of Tier 1 apps have been virtualized by other companies
- Whether the virtualization project is a stepping stone towards the cloud
- What challenges are preventing people from finishing (or even starting) virtualization of Tier 1 apps

The survey is only 10 questions long–and again, entering gives you the possibility of winning an iPad, as well as a guaranteed copy of the survey results.

Take the survey now!

We look forward to reviewing your responses!

application, application development, application performance management, cloud computing, virtualization

This is the first post in a series on the Top Application Performance Challenges.

Of the many issues affecting the performance of Java applications, synchronization ranks near the top.  Issues arising from synchronization are often hard to recognize and their impact on performance can be become significant. What’s more, they are often, at least in principle, avoidable.

The fundamental need to synchronize lies with Java’s support for concurrency. This is implemented by allowing the execution of code by separate threads within the same process. Separate threads can share the same resources, objects in memory. While being a very efficient way to get more work done (while one thread waits for an IO operation to complete, another thread gets the CPU to run a computation), the application is also exposed to interference and consistency problems.

The JVM does not guarantee an execution order of the code running in concurrent threads. If multiple threads reference the same object there is no telling what state that object will be in at a given moment in time. The repercussions of that simple fact can be enormous with, for example, one thread running calculations and returning wrong results because a concurrent thread accessing and modifying shared bits of information at the same time.

To prevent such a scenario (a program needs to execute correctly, after all) a programmer uses the “synchronize” keyword in his/her program to force order on concurrent thread execution. Using “synchronize” prevents threads from obtaining the same object at the same time.

In practice, however, this simple mechanism comes with substantial side effects. Modern business applications are typically highly multi-threaded. Many threads execute concurrently, and consequently “contend” heavily for shared objects. Contention occurs when a thread wants to access a synchronized object that is already held by another thread. All threads contending effectively “block,” halting their execution until they can acquire the object. Synchronization effectively forces concurrent processing back into sequential execution.

With just a few metrics we can show the effects of synchronization on an application’s performance. For instance, take a look at the graph below.

While increasing load (number of users = blue), we see that at some point midway the response time (yellow) takes an upward curve, while at the same time resource usage (CPU = red) somewhat increases to eventually plateau and even recedes. It almost looks like the application runs with the “handbrake on,” a classic, albeit high-level, symptom of an application that has been “over-synchronized.”

With every new version of the JVM improvements are made to mitigate this issue. However, while helpful, these improvements can’t fully resolve the issue and address the application’s negative performance.

Also, developers have come to adopt “defensive” coding practices, synchronizing large pieces of code to prevent possible problems. In large development organizations this problem is further magnified as no one developer or team has full ownership of an application’s entire code base. The practice to err on the side of safety can quickly exacerbate with large portions of synchronized code significantly impacting the performance of an application’s potential throughput.

It is often too arduous a task to maintain a locking strategy fine grained enough to ensure that only the necessary minimum of execution paths are synchronized. New approaches to better manage state in a concurrent environment are available in newer versions of Java such as readWriteLocks, but they are not widely adopted yet.  These approaches promise a higher degree of concurrency, but it will always be up to the developer to implement and use the mechanism correctly.

Is synchronization, then, always going to result in a high MTTR?

New technologies exist on the horizon that may lend some relief.  Software Transactional Memory Systems (STM), for example, might become a powerful weapon for dealing with synchronization issues. They may not be ready for prime time yet, but given what we’ve seen with database systems, they might be the key to taming the concurrency challenges affecting applications today. Check out JVSTM, Multiverse and Clojure for examples of STMs.

For now, the best development organizations are the ones that can walk the fine line of balancing code review/rewrite burdens and concessions to performance. APM tools can help quite a lot in such scenarios, allowing to monitor application execution under high load (aka “in production”) and quickly pinpoint to the execution times for particular highly contended Objects, Database connections being a prime example. With the right APM in place, the ability to identify thread synchronization issues become greatly increased—and the overall MTTR will drop dramatically.

application development, application performance management, developers, synchronization, thread

I’ve never written a blog post before, but Jyoti tells me, from experience, that if you write about your passion, it’s a piece of cake. So here goes. My passion is to create technology to make application management easier and more productive so that companies can focus on delivering value instead of fretting about just being able to run apps 24×7. Since I intend to keep up this newly acquired habit of mine, let me start by establishing the fundamental tenet on which this technology we build at AppDynamics is based – Business Transactions.

You may think you’ve heard this discussion before, but I’d wager a guess that you’ve actually heard more about transactions than about business. I want to tell you why you actually need to put the focus on the business.

In today’s world where more and more businesses move to the web, the application is the face of the company. It is the business, it is the revenue stream. From DVD rentals to talent management, everything is an online application. Someone suggested to me recently that the only business they cannot use the internet for was getting a haircut. But I am sure they are working on it too!!

From the CIO to the ops team pushing the latest application release out, there needs to be a common context that drives the business unit towards a robust, efficient and a highly competitive application. That context is the business transaction.

Here are a few reasons why you should focus on the ‘business’ side of the business transaction:

1. When the application is the business, competitive advantage means having a better application. Now in an ideal world, when someone builds an application they would like it to be better than anything out there. But having an edge is not a one time thing. It’s constant evolution, which means rapid change and faster adaptation (Flickr does 10+ deployments a day!!). Dynamic changes to an application’s features and functions directly impact the business transaction and the overall user experience. At the end of the day, what you care about most is how your users are being serviced.

2. When the application is the business, managing an application is no longer just about monitoring CPU, JVM memory or timing key methods. It’s really about understanding the user experience, managing the business operations and creating service levels to ensure optimum performance. Business transactions are the binding factor for attaching SLAs to business operations and for creating a common ground between dev teams and ops teams. Focusing on the business transaction is a great way to validate the whole cycle including a rollout or the current state of the app at any given time.  Also, the growth of business means more capacity to serve. This means the growth of servers. So, instead of multiplying management complexity with more resources, a focus on user-centric SLAs is the only scalable way to address performance.

3. When the application is the business, the responsibility of managing applications (and hence the revenue stream) falls on IT Operations. But development is responsible for building the app and for the innovation in it. The creates an interdependence (and what we see as the DevOps movement). By creating a common language to categorize user requests – “Are the checkouts doing ok?” “Is the sales order processing running faster than before?” – these teams are better able to communicate and effectively manage application performance over time.

Using the business transaction as a unit of management for the application makes your company more agile, competitive, scalable and high performance. In future posts, I’ll review the various ways to approach overall application management and take a deeper dive on why a business transaction focus is critical to success.

At the end of the day, it should be all about the business!

appdynamics, application, application monitoring, application performance management, DevOps

Last week we hosted a presentation by AppDynamics customer, Priceline, along with an informative customer roundtable on application performance. During our roundtable, we polled the audience on their challenges with application performance – we asked questions about their architectures, development philosophies, common performance problems, downtime and visibility. Roughly 100 attendees from companies big and small responded.

The results were interesting. Here’s a look at what we found:

When asked about performance challenges, 88% of respondents noted they’ve experienced application performance problems in the last 12 months. The most common performance issues cited were Slow Response, Stalls, Errors and Memory Leaks. (Stay tuned this August for our upcoming blog series on what you should know about the Top 5 App Performance Challenges).

Additionally, 67% of respondents said they struggled to determine the root cause of their application performance problem in a timely manner, and roughly 33% responded that end-users called the helpdesk to complain about poor online experiences. Surprisingly, only 42% of participants admitted that they’ve experienced production downtime.

When asked to share some information about their application environments, participants responded that:

• 73% have a multi-tier, distributed application
• 70% leverage both open source and commercial application infrastructure (i.e. app servers)
• 52% say they follow an agile development philosophy
• 82% said that they lack visibility in key areas of their application

For the most part, we weren’t surprised by these answers. We hear from our customers everyday that they are shifting toward SOA, virtualization and cloud computing infrastructures. These responses also confirmed our experience that many organizations have built JBoss and Tomcat into their application architectures, whereas five years ago most companies would primarily have had WebLogic or WebSphere.

Its also not surprising that such a large number of respondents had experienced issues with performance. Distributed applications, especially those that use a mix of open source and proprietary components, are extremely difficult to monitor and troubleshoot. They often contain blind spots that result in critical performance problems.

Nor was it surprising that the majority of companies lack visibility into their applications. This is the most common refrain we hear when talking to companies for the first time. It doesn’t matter if they are running a $10 billion dollar business or a $10 million dollar business…visibility is always a concern. At the end of the day, “You can’t manage what you can’t see.”

Management really is the kicker. Web-applications have become transaction-oriented and revenue-critical, and it is more important than ever to be able to not only monitor, but quickly diagnose and fix performance issues before they impact your business.

appdynamics, application development, application monitoring, application performance management, cloud computing

I recently had a chance to visit the webcast “This Week in Cloud Computing” and share some of my thoughts about cloud trends and application performance management. One thread of the conversation that I found particularly interesting was the discussion of agility in cloud computing. Although this theme comes up from time to time, most discussions I hear on cloud computing focus on cost-cutting and security.

These are extremely important concerns, of course — security in particular can be seen as a prerequisite of any sound cloud computing strategy. But there’s a “forest for the trees” risk in focusing too much on cloud computing pitfalls in lieu of recognizing its benefits, of which agility is certainly a major component.

We’re seeing with our own customers the need to be even more agile than before, of scrums becoming common and engineering stand-ups becoming a way of life. Any process change that helps speed up the application deployment chain is more than just a “nice to have;” it’s a sea change in the ability of companies to deliver value to their end users.

Bernard Golden makes some interesting points about two types of cloud computing agility in this discussion on CIO – definitely worth a read if you’re interested in the topic.

In case you missed the live webcast last week, here’s the video:

application development, application performance management, cloud computing, hybrid clouds, public clouds

AppDynamics recently had the privilege of speaking at the first U.S.-based DevOps event, held at the LinkedIn corporation in Santa Clara. We had some web operations rock stars on hand–including John Allspaw from Etsy, John Willis from Opscode, and Patrick Debois, father of the DevOps movement.

But perhaps the true stars of the event were the 200+ operations and development people in the audience–because it was their Tweets, questions, and ongoing feedback during the event that made it so much fun. Therefore, presented without further commentary, is our personal list of the best quotes from Dev Ops Days:

“If it isn’t monitored, it isn’t production!”

“In Ops you can never exceed expectations, because the expectation is 100% up-time.”

“No matter how careful or good you are, sh!t will happen.”

“If you’re in Ops – you better have metrics. When in doubt – make *&#!@# graphs!”

“Traditional metrics of cpu and memory usage don’t matter to your customers. How ’bout measuring what really matters to your customers?”

“Whether you use ITIL or not is not important – what matters is if you can measure if you are improving!”

“In most organizations, Dev and Ops have misaligned goals. Dev is measured by the number of new features. Ops is measured by 100% uptime. Question: What’s the best way to get 100% uptime? Answer: Don’t introduce any new features or make any changes.”

“The relationship between dev and ops in a company is defined by the release process. You will understand the relationship if you examine this process.”

“If Ops goes to lunch with Ops and Dev goes to lunch with Dev, a low level of efficiency is a certainty.”

“Lack of trust in an organization is really expensive. You can’t villianize others if you know their kids.”

“Ops now has the same pride in their tools that Dev has always had.”

“One part of DevOps is bringing Dev culture and tools into Ops – version control, testing tools, automation tools, and repeatibility with “observability” ”

“Limit your liability by making very small incremental changes.”

“Fail quick, fail often, recover quickly.”

“Lots of focus at web 2.0 startups on new features…but very little focus on the “abilities” – scalability, flexibility, durability.”

“What you need to do is hire smart people and give them root. @#$% Yeah!”

“Even if your organization is not agile, release all the time so that you are experienced when it matters.”

“Unplanned work steals time from planned work.”

application development, DevOps, IT operations