Memory, Memory, Memory…

Memory is a critical part of Java, in particular, the management of memory. As a developer, memory management is not something you want to be doing on a regular basis, nor is it something you want to do manually. One of the great benefits of Java is its ability to take care of the memory model for you. When objects aren’t in use, Java helps you out by doing the clean up.

Read the Full Post…

APM Thought Leadership, application development, application monitoring, Java, memory leaks, memory thrash

It’s time for another update to our series on Top Application Performance Challenges. Last time I looked at Java’s synchronization mechanism as a source for performance problems. This time around I take on what is likely the Performance Engineer’s bread and butter … slow database access!

Behind this small statement lies a tricky and multifaceted discussion. For now, I’m going to focus on just one particular aspect – the Object Relational Mapper (ORM).

The ORM has become a method of choice for bringing together the two foundational technologies that we base business applications on today – object-oriented applications (Java, .NET) and relational databases (Oracle, mySQL, PostgreSQL, etc.). For many developers, this technology can seem like a godsend, eliminating the need for them to drill-down into the intricacies of how these two technologies interact. But at the same time, ORMs can place an additional burden on applications, significantly impacting performance while everything looks fine on the surface.

Here’s my two cents on ORMs and why developers should take a longer look under the hood:

In the majority of cases, the time and resources taken to retrieve data are orders of magnitude greater than what’s required to process it. It is no surprise that performance considerations should always include the means and ways of accessing and storing data.

I already mentioned the two major technology foundations on which we build business applications today, object oriented applications, used to model and execute the business logic, and relational databases, used to manage and store the data. Object oriented programs unite data and logic into object instances, relational databases on the other hand isolate data into columns and tables, joined by keys and indexes.

This leaves a pretty big gap to bridge, and it falls upon the application to do the legwork. Since bridging this gap is something many applications must do, enter the ORM as a convenient, reusable framework. Hibernate, for instance, is quite likely the most popular ORM out there.

While intuitive for an application developer to use (ORMs do hide the translation complexities) an ORM can also be a significant weight on an application’s performance.

Let me explain.

Take a Call Graph from AppDynamics and follow the execution path of a transaction, method by method, from the moment a user request hits the application until calls to the database are issued to retrieve the requested information. Then size up the layers of code this path has to go through to get to the data. If your application has implemented an ORM like Hibernate, I assure you’ll be surprised how much stuff is actually going on in there.

No finger-pointing intended. A developer will emphasize that the benefits of just using the ORM component (without having to understand how its working) greatly increases his productivity. Point taken. It does pay, however, to review an ORM’s data access strategy.

I recently worked with a company and saw transactions (single user requests) that each bombard the database with 3000+ distinct calls. Seems a little excessive? You would be right. More over, nobody knew that this was happening and certainly nobody intended for it to be so.

In many cases simple configuration settings or using a different ‘fetch’ method offered by the ORM itself can affect performance significantly. Whether for instance the ORM accesses each row of the customer table individually to fill an array of customers in your code or is actually constructing a query that encompasses all of the expected result-set and gets them in one fell swoop does make a big difference.

Seems obvious, right? But do you actually know what your ORM really does when retrieving the information?

You might be surprised.

application development, application monitoring, database calls, developers, slow SQL

This summer, I had a chance to sit on a panel at DevOps Days with a few industry colleagues to discuss the role of monitoring, testing, and performance in solving DevOps issues. It was a lively discussion on the teams, tools, and techniques that play a role in managing application and network performance.

Read the Full Post…

APM Thought Leadership, application development, DevOps, IT operations

Interested in winning an iPad? Take our Application Virtualization survey, and we’ll give you a more-than-decent shot at winning your very own!

In talking to our customers, AppDynamics recognizes that virtualizing mission-critical applications is at the top of everyone’s mind, even though some companies are at different stages of the process than others.

Many IT departments have already finished virtualizing their less critical apps and they realize that there are many benefits to be had in virtualizing the rest. But, in many cases, application owners are concerned about performance impacts to their mission-critical apps, and they’re saying, “Hands off my app.”

What’s it like at your organization? Is the path to app virtualization a smooth one, or are you experiencing some bumps along the way? We’d like to find out more, and we’d like your help to do it.

Answer a few short questions on your app virtualization strategy and be automatically entered to win an iPad. In addition, we’ll send you a copy of the results, so you can discover what your peers at other organizations are doing.

Find out:

- What percentage of Tier 1 apps have been virtualized by other companies
- Whether the virtualization project is a stepping stone towards the cloud
- What challenges are preventing people from finishing (or even starting) virtualization of Tier 1 apps

The survey is only 10 questions long–and again, entering gives you the possibility of winning an iPad, as well as a guaranteed copy of the survey results.

Take the survey now!

We look forward to reviewing your responses!

APM Thought Leadership, application, application development, cloud computing, virtualization

This is the first post in a series on the Top Application Performance Challenges.

Of the many issues affecting the performance of Java/.NET applications, synchronization ranks near the top.  Issues arising from synchronization are often hard to recognize and their impact on performance can be become significant. What’s more, they are often, at least in principle, avoidable.

The fundamental need to synchronize lies with Java’s support for concurrency. This is implemented by allowing the execution of code by separate threads within the same process. Separate threads can share the same resources, objects in memory. While being a very efficient way to get more work done (while one thread waits for an IO operation to complete, another thread gets the CPU to run a computation), the application is also exposed to interference and consistency problems.

The JVM/CLR does not guarantee an execution order of the code running in concurrent threads. If multiple threads reference the same object there is no telling what state that object will be in at a given moment in time. The repercussions of that simple fact can be enormous with, for example, one thread running calculations and returning wrong results because a concurrent thread accessing and modifying shared bits of information at the same time.

To prevent such a scenario (a program needs to execute correctly, after all) a programmer uses the “synchronize” keyword in his/her program to force order on concurrent thread execution. Using “synchronize” prevents threads from obtaining the same object at the same time.

In practice, however, this simple mechanism comes with substantial side effects. Modern business applications are typically highly multi-threaded. Many threads execute concurrently, and consequently “contend” heavily for shared objects. Contention occurs when a thread wants to access a synchronized object that is already held by another thread. All threads contending effectively “block,” halting their execution until they can acquire the object. Synchronization effectively forces concurrent processing back into sequential execution.

With just a few metrics we can show the effects of synchronization on an application’s performance. For instance, take a look at the graph below.

While increasing load (number of users = blue), we see that at some point midway the response time (yellow) takes an upward curve, while at the same time resource usage (CPU = red) somewhat increases to eventually plateau and even recedes. It almost looks like the application runs with the “handbrake on,” a classic, albeit high-level, symptom of an application that has been “over-synchronized.”

With every new version of the JVM/CLR improvements are made to mitigate this issue. However, while helpful, these improvements can’t fully resolve the issue and address the application’s negative performance.

Also, developers have come to adopt “defensive” coding practices, synchronizing large pieces of code to prevent possible problems. In large development organizations this problem is further magnified as no one developer or team has full ownership of an application’s entire code base. The practice to err on the side of safety can quickly exacerbate with large portions of synchronized code significantly impacting the performance of an application’s potential throughput.

It is often too arduous a task to maintain a locking strategy fine grained enough to ensure that only the necessary minimum of execution paths are synchronized. New approaches to better manage state in a concurrent environment are available in newer versions of Java such as readWriteLocks, but they are not widely adopted yet.  These approaches promise a higher degree of concurrency, but it will always be up to the developer to implement and use the mechanism correctly.

Is synchronization, then, always going to result in a high MTTR?

New technologies exist on the horizon that may lend some relief.  Software Transactional Memory Systems (STM), for example, might become a powerful weapon for dealing with synchronization issues. They may not be ready for prime time yet, but given what we’ve seen with database systems, they might be the key to taming the concurrency challenges affecting applications today. Check out JVSTM, Multiverse and Clojure for examples of STMs.

For now, the best development organizations are the ones that can walk the fine line of balancing code review/rewrite burdens and concessions to performance. APM tools can help quite a lot in such scenarios, allowing to monitor application execution under high load (aka “in production”) and quickly pinpoint to the execution times for particular highly contended Objects, Database connections being a prime example. With the right APM in place, the ability to identify thread synchronization issues become greatly increased—and the overall MTTR will drop dramatically.

APM Thought Leadership, application development, developers, synchronization, thread

Last week we hosted a presentation by AppDynamics customer, Priceline, along with an informative customer roundtable on application performance. During our roundtable, we polled the audience on their challenges with application performance – we asked questions about their architectures, development philosophies, common performance problems, downtime and visibility. Roughly 100 attendees from companies big and small responded.

The results were interesting. Here’s a look at what we found:

When asked about performance challenges, 88% of respondents noted they’ve experienced application performance problems in the last 12 months. The most common performance issues cited were Slow Response, Stalls, Errors and Memory Leaks. (Stay tuned this August for our upcoming blog series on what you should know about the Top 5 App Performance Challenges).

Additionally, 67% of respondents said they struggled to determine the root cause of their application performance problem in a timely manner, and roughly 33% responded that end-users called the helpdesk to complain about poor online experiences. Surprisingly, only 42% of participants admitted that they’ve experienced production downtime.

When asked to share some information about their application environments, participants responded that:

• 73% have a multi-tier, distributed application
• 70% leverage both open source and commercial application infrastructure (i.e. app servers)
• 52% say they follow an agile development philosophy
• 82% said that they lack visibility in key areas of their application

For the most part, we weren’t surprised by these answers. We hear from our customers everyday that they are shifting toward SOA, virtualization and cloud computing infrastructures. These responses also confirmed our experience that many organizations have built JBoss and Tomcat into their application architectures, whereas five years ago most companies would primarily have had WebLogic or WebSphere.

Its also not surprising that such a large number of respondents had experienced issues with performance. Distributed applications, especially those that use a mix of open source and proprietary components, are extremely difficult to monitor and troubleshoot. They often contain blind spots that result in critical performance problems.

Nor was it surprising that the majority of companies lack visibility into their applications. This is the most common refrain we hear when talking to companies for the first time. It doesn’t matter if they are running a $10 billion dollar business or a $10 million dollar business…visibility is always a concern. At the end of the day, “You can’t manage what you can’t see.”

Management really is the kicker. Web-applications have become transaction-oriented and revenue-critical, and it is more important than ever to be able to not only monitor, but quickly diagnose and fix performance issues before they impact your business.

APM Thought Leadership, appdynamics, application development, application monitoring, cloud computing

I recently had a chance to visit the webcast “This Week in Cloud Computing” and share some of my thoughts about cloud trends and application performance management. One thread of the conversation that I found particularly interesting was the discussion of agility in cloud computing. Although this theme comes up from time to time, most discussions I hear on cloud computing focus on cost-cutting and security.

These are extremely important concerns, of course — security in particular can be seen as a prerequisite of any sound cloud computing strategy. But there’s a “forest for the trees” risk in focusing too much on cloud computing pitfalls in lieu of recognizing its benefits, of which agility is certainly a major component.

We’re seeing with our own customers the need to be even more agile than before, of scrums becoming common and engineering stand-ups becoming a way of life. Any process change that helps speed up the application deployment chain is more than just a “nice to have;” it’s a sea change in the ability of companies to deliver value to their end users.

Bernard Golden makes some interesting points about two types of cloud computing agility in this discussion on CIO – definitely worth a read if you’re interested in the topic.

In case you missed the live webcast last week, here’s the video:

APM Thought Leadership, application development, cloud computing, hybrid clouds, public clouds

AppDynamics recently had the privilege of speaking at the first U.S.-based DevOps event, held at the LinkedIn corporation in Santa Clara. We had some web operations rock stars on hand–including John Allspaw from Etsy, John Willis from Opscode, and Patrick Debois, father of the DevOps movement.

But perhaps the true stars of the event were the 200+ operations and development people in the audience–because it was their Tweets, questions, and ongoing feedback during the event that made it so much fun. Therefore, presented without further commentary, is our personal list of the best quotes from Dev Ops Days:

“If it isn’t monitored, it isn’t production!”

“In Ops you can never exceed expectations, because the expectation is 100% up-time.”

“No matter how careful or good you are, sh!t will happen.”

“If you’re in Ops – you better have metrics. When in doubt – make *&#!@# graphs!”

“Traditional metrics of cpu and memory usage don’t matter to your customers. How ’bout measuring what really matters to your customers?”

“Whether you use ITIL or not is not important – what matters is if you can measure if you are improving!”

“In most organizations, Dev and Ops have misaligned goals. Dev is measured by the number of new features. Ops is measured by 100% uptime. Question: What’s the best way to get 100% uptime? Answer: Don’t introduce any new features or make any changes.”

“The relationship between dev and ops in a company is defined by the release process. You will understand the relationship if you examine this process.”

“If Ops goes to lunch with Ops and Dev goes to lunch with Dev, a low level of efficiency is a certainty.”

“Lack of trust in an organization is really expensive. You can’t villianize others if you know their kids.”

“Ops now has the same pride in their tools that Dev has always had.”

“One part of DevOps is bringing Dev culture and tools into Ops – version control, testing tools, automation tools, and repeatibility with “observability” ”

“Limit your liability by making very small incremental changes.”

“Fail quick, fail often, recover quickly.”

“Lots of focus at web 2.0 startups on new features…but very little focus on the “abilities” – scalability, flexibility, durability.”

“What you need to do is hire smart people and give them root. @#$% Yeah!”

“Even if your organization is not agile, release all the time so that you are experienced when it matters.”

“Unplanned work steals time from planned work.”

application development, DevOps, IT operations

I haven’t written a blog post before, but I’m told that writing is easy if you write about your passions. So let me discuss why I founded AppDynamics.

I began the company because I saw a gap that I wanted to fill. The world is host to countless thousands of applications, and an almost equal number of people wanting to help IT professionals manage those applications. But somehow, no one ever made managing application performance easy. It was always difficult, complex, and costly.

In addition, most application performance companies were quickly falling behind the times. You may have heard the expression that “nothing dates like science fiction”–it’s easy to look at a science fiction film and instantly know when it was made. Similarly, application performance solutions quickly lose their luster if they’re not constantly adapting to the changing IT environment. What I knew was that distributed applications were becoming more, not less common–and that SOA, virtualization, and the cloud were becoming ever prevalent. Traditional monitoring companies couldn’t keep up with the demands of these new environments.

I wanted to start a company that brought relief to IT professionals who were still on the front lines, managing application performance, but no longer had tools equipped to do the job. I wanted to make managing a complex application as easy as reading a Google traffic map. I wanted to give them the ability to not only monitor application performance, but also find and fix root-cause problems. I wanted to combine traditional monitoring functions with the ability to orchestrate capacity in the cloud.

I won’t use this blog space to only talk about what my company does–I expect to discuss trends in the industry, issues facing IT operations and developers, and other topics that come to mind. But I will always circle back to the bottom line–simplifying the lives of IT professionals by making application performance easy.

Because if you’re supposed to write about your passion, then that is my passion.

appdynamics, application, application development, DevOps