AppDynamics is spending the week in Las Vegas–or, as our Southwest pilot wisecracked as our plane descended, “Lost Wages.” But it’s not for fun—at least, not that kind of fun. We’re talking to prospects and customers at the 2010 Gartner Data Center conference.

Read the Full Post…

cloud computing, infrastructure, IT operations, virtualization

San Francisco’s QCon was expecting a smaller crowd, but ended up bursting at the seams: the event sold out weeks ahead of time and in many sessions it was standing room only.

Targeted at architects and operations folks as much as developers, QCon was heavy on the hot topics of the day: SOA, agile, and DevOps. But if there was a consistent trend throughout the three days, it was “No more theory. Show us the practice.”

At Jesper Boeg’s talk for example—“Raising the Bar: Using Kanban and Lean to Super Optimize Your Agile Implementation”—the talk was peppered with some good sound bites (“If it hurts, do it more often and bring the pain forward”). But it also stressed the meat: Boeg demonstrated a “deployment pipeline” that represented an automated implementation of the build, deploy, test, and release process—a way to find and eliminate bottlenecks in agile delivery.

Similarly, John Allspaw started high in his talk—sharing his ideas on the areas of ownership and specialization between Ops and Dev, a typical DevOps presentation—but backs up the theory with code-level discussions of how logging, metrics, and monitoring works at Etsy.  (His blog entry on the subject and complete Qcon slides can be found on his blog, Kitchen Soap.)

Adrian Cockroft, who is leading a trailblazing public cloud deployment of production-level applications at Netflix, also wrapped theory around juicy substance. He “showed the code” and the screenshots of his company’s app scaling and monitoring tools (you can find his complete slide presentation here).

Not everyone took the time to drill down, though. Tweets from QCon attendees showed that the natives got restless in talks that stayed too high level:

“OK, just because you can draw a block diagram out of something doesn’t mean it makes sense.”

“Ok, we get it. Your company is very interesting, now get to the nerd stuff.”

“These sessions are high-level narratives. Show me the code, guys! Devil’s in the details.”

At the same time, they would shower plaudits and congratulations on speakers who gave them what they wanted: something new to learn.

When the Twitter stream started to compare QCon’s activities with an event happening concurrently in the city, Cloud Expo, the nature of the attendees was draw into sharp relief:

“At #cloudexpo people used laptops during sessions to check email… At #qconsf they are writing code.”

When it comes to agile, SOA, DevOps, and other problems of the day, people are ready for answers.

APM Thought Leadership, cloud computing, developers, DevOps, IT operations

AppDynamics is founded on a set of deeply held beliefs regarding our industry and how it’s changed over the last several years.  But it’s never good to let deeply held beliefs stay unchallenged.  So every now and then, we do a reality check.

Our most recent reality check was during our webinar presentation of AppDynamics 3.0. We attracted hundreds of IT ops and dev professionals who wanted to learn both about our solution as well as the specific features of our new release—so we took the opportunity to poll them and ask them a few questions. First, we asked if they operated in a SOA environment:

AppDynamics believes that applications are increasingly moving to SOA, turning monolithic web architectures from the early 2000s into obsolete antiques.  As you can see, that belief was confirmed; the vast majority of our webinar attendees have already entered that world.

Then we asked if they followed an agile development approach:

Again, the vast majority of attendees have embraced agile—in fact, nearly 50% release new features or capabilities at least monthly!  Only 8% report that they follow the traditional, waterfall approach to development. With those kinds of tumultuous deadlines, AppDynamics remains impressed that these hardy souls were actually able to take enough time out of their schedule to watch our presentation.

Finally, we wanted to know the punch line: what’s the effect of all this change on their ability to manage application performance?

Over half were really feeling the crunch, and only a scant few had escaped unscathed.

It’s not that AppDynamics enjoys the pain of others. (We don’t. Really.)  But having our fundamental beliefs confirmed—that the world of applications has changed, and application management solutions need to change with them—simply lets us know that we exist for the right reasons.

Take the example of one of our most recent customers, TiVo.  Operating in a highly distributed environment, TiVo has hundreds of individual Java and proprietary applications, designed to work together to deliver service to its customers.

“We used to spend hours troubleshooting issues,” Richard Rothschild, Senior Director of IT, told us.  “If a service was running slowly and we didn’t know the cause, finding that root cause was like looking for a needle in a haystack.

He continued, “We used to spend up to 6 hours on root cause. AppDynamics brought that time down to ten minutes.  We’ve already seen a big improvement in the reliability and uptime of our services—anything that simplifies our job in this complex environment makes us feel much more confident about taking on new business projects.”

It’s complicated out there, and with the advent of cloud and virtual environments, it’s not getting any easier. But we went into this business in order to simplify application performance management and support application teams in their quest for both performance and availability.  So far, it looks like we chose the right reasons to exist.

APM Thought Leadership, appdynamics, cloud computing, IT operations, SOA

This summer, I had a chance to sit on a panel at DevOps Days with a few industry colleagues to discuss the role of monitoring, testing, and performance in solving DevOps issues. It was a lively discussion on the teams, tools, and techniques that play a role in managing application and network performance.

Read the Full Post…

APM Thought Leadership, application development, DevOps, IT operations

This post is part of our series on the Top Application Performance Challenges.

Over the past couple of weeks we helped a few DevOps groups troubleshoot performance problems in their production Java-based applications. While the applications themselves bore little resemblance to one another, they all shared one common problem. They were full of runtime exceptions.

When I brought the exceptions to the attention of some of the teams I got a variety of different answers.

• “Oh yeah, we know all about that one, just some old debug info … I think”
• “Can you just tell the APM tool to just ignore that exception because it happens way too much”
• “I have no idea where that exception comes from, I think it’s the vendor/contractor/off shore team’s/someone else’s code”
• “Sometimes that exception indicates an error, sometimes we use it for flow control”

The response was the same – because the exception did not affect the functionality of (read: did not break) the application it was deemed unimportant. But what about performance?

In one high-volume production application (100’s of calls/second) we observed an NPE (Null Pointer Exception) rate of more than 1000 exceptions per minute (17,800/15 min). This particular NPE occurred in the exact same line of code and propagated some 20 calls up in the stack before a catch clause handled it. The pseudo code for the line read if the string equaled null and the length of the trimmed string was zero then do x otherwise just exit. When we read the line out loud the problem was immediately obvious – you cannot trim a null string. While the team agreed that they needed to fix the code, they remained skeptical that simply changing ‘=’ to ‘!’ would really improve performance.

Using basic simulations we estimate this error rate imposes a 3-4% slow down on basic response time at a minimum. You heard me, 3-4% slow down at a minimum.

And these response issues get compounded in multi-threaded application server environments. When multiple pooled threads slow down to cope with error handling then the application has fewer resources available to process new requests. At the JVM level the errors cause more IO, more objects created in the heap, more garbage collection. In a multi-threaded environment the error threatens more than response times, it threatens load capacity.

And if the application is distributed across multiple JVM’s … well you get the picture.

The web contains many articles and discussions on the performance impact of Java’s exception handling. The examples in these posts provide sufficient data to show that a real performance penalty exists for throwing and catching exceptions. Doing the fix really does improve the performance!

Runtime Exceptions happen. When they occur frequently, they do appreciably slow down your application. The slowness becomes contagious to all transactions being served by the application. Don’t mute them. Don’t ignore them. Don’t dismiss them. Don’t convince yourself they are harmless. If you want a simple way to improve your applications performance, start by fixing up these regular occurring exceptions. Who knows, you just might help everyone’s code run a little faster.

APM Thought Leadership, application, application monitoring, developers, DevOps, IT operations, runtime exception, slow response error

AppDynamics recently had the privilege of speaking at the first U.S.-based DevOps event, held at the LinkedIn corporation in Santa Clara. We had some web operations rock stars on hand–including John Allspaw from Etsy, John Willis from Opscode, and Patrick Debois, father of the DevOps movement.

But perhaps the true stars of the event were the 200+ operations and development people in the audience–because it was their Tweets, questions, and ongoing feedback during the event that made it so much fun. Therefore, presented without further commentary, is our personal list of the best quotes from Dev Ops Days:

“If it isn’t monitored, it isn’t production!”

“In Ops you can never exceed expectations, because the expectation is 100% up-time.”

“No matter how careful or good you are, sh!t will happen.”

“If you’re in Ops – you better have metrics. When in doubt – make *&#!@# graphs!”

“Traditional metrics of cpu and memory usage don’t matter to your customers. How ’bout measuring what really matters to your customers?”

“Whether you use ITIL or not is not important – what matters is if you can measure if you are improving!”

“In most organizations, Dev and Ops have misaligned goals. Dev is measured by the number of new features. Ops is measured by 100% uptime. Question: What’s the best way to get 100% uptime? Answer: Don’t introduce any new features or make any changes.”

“The relationship between dev and ops in a company is defined by the release process. You will understand the relationship if you examine this process.”

“If Ops goes to lunch with Ops and Dev goes to lunch with Dev, a low level of efficiency is a certainty.”

“Lack of trust in an organization is really expensive. You can’t villianize others if you know their kids.”

“Ops now has the same pride in their tools that Dev has always had.”

“One part of DevOps is bringing Dev culture and tools into Ops – version control, testing tools, automation tools, and repeatibility with “observability” ”

“Limit your liability by making very small incremental changes.”

“Fail quick, fail often, recover quickly.”

“Lots of focus at web 2.0 startups on new features…but very little focus on the “abilities” – scalability, flexibility, durability.”

“What you need to do is hire smart people and give them root. @#$% Yeah!”

“Even if your organization is not agile, release all the time so that you are experienced when it matters.”

“Unplanned work steals time from planned work.”

application development, DevOps, IT operations

The debate over who owns an application in production continues to unfurl.  What I’ve found interesting is, after a period of time where writers and bloggers were looking towards IT Operations to be application owners, I’ve started to detect a bit of a backlash.

The arguments for IT Operations owning the application is simple:

• Development should stay focused on creating new applications and adding features, not maintaining what’s already in production
• If you ask a developer, they’ll always say they’d rather spend their time on innovation, rather than production support or bug fixes
• IT Operations generally oversees the production infrastructure (servers, storage, networks etc), so they are the natural caretakers of production applications as well

Analysts often point out that this is easier said that done: what is sometimes called the “required cooperation” between operations and development is often difficult to obtain.  I’ve also seen it suggested that putting production applications in the hands of operations is a Utopian dream, leading to performance issues and SLA violations.

If I were to describe my own view of “natural selection” in regards to managing application performance, it would be more along the lines of collaboration. The development team is likely to help support applications as long as IT Operations is able to provide them with the information and data they need to fix root-cause issues quickly.  Because much development is now done in agile environments, this sort of teamwork is becoming less of a philosophical choice and more of a business necessity.

If you look through blogs and Twitter, you’ll find some interesting grassroots movements such as #devops and #agileoperations.  These are communities forming that acknowledge the need to break down the traditional walls that exist between Dev and Ops, and radically restructure those relationships so that they are focused on shared goals and outcomes.

One devops proponent, James Turnbull at Kartar.net, explains the problem:

“So … why should we merge or bring together the two realms?  Well there are lots of reasons but first and foremost because what we’re doing now is broken.  Really, really broken.  In many shops the relationship between development (or engineering) and operations is dysfunctional to the point of occasional toxicity.”

(I love the phrase “occasional toxicity”…)

He goes on to add:

“DevOps is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups. In a DevOps environment, developers and sysadmins build relationships, processes, and tools that allow them to better interact and ultimately better service the customer.

“DevOps is also more than just software deployment – it’s a whole new way of thinking about cooperation and coordination between the people who make the software and the people who run it.  Areas like automation, monitoring, capacity planning & performance, backup & recovery, security, networking and provisioning can all benefit from using a DevOps model to enhance the nature and quality of interactions between development and operations teams.”

I believe that the question that comes naturally out of these conversations is this: does IT operations have the tools they need to facilitate this collaboration with their peers in development?  Traditionally, they haven’t.  The tools either didn’t provide much deep visibility into the application, or when they did provide deep visibility, they were extremely complicated for IT Operations to be able to understand and use. But creating those tools, and encouraging that collaboration, is one of my own company’s guiding principles.

Applications are becoming more complex and distributed, and development is increasingly taking place in the context of agile release cycles.  So really, the question isn’t “who owns the app”–but how best to foster the collaborative process that enables dev and ops to both build out applications and resolve their performance problems, and to do so in record time.

APM Thought Leadership, application, application monitoring, developers, DevOps, IT operations