TAG | automated remediation
I recently had the luxury of sitting on a beach for a while to unwind, de-stress, and recharge my batteries. Even though I really shouldn’t have been thinking about work, the IT performance geek inside of me is always lurking. He can’t be switched off so I allow him to enter the picture, run away with my thoughts for a while, and then lock him away until the next time he breaks out into the wild. This blog post is about one interesting observation the IT performance geek made while I was watching crabs go about their business on the beach.
On my first day of vacation I saw a bunch of holes in the beach with mounds of sand piled up outside of each hole. I thought this to be quite interesting and wondered what sort of creature was the excavating genius. After some time passed by I noticed some suspiciously crabby eyes peering out of the holes at me. If I made any moves the crabs would dart back deep into their burrows and let some time pass until they peered outside again.
Later that day the tide came in and washed away the sand piles covering up the holes that the crabs had made. I wondered what happened to them while the tide was up. Did they stay under the sand and water? Did they go somewhere else? Did they rebuild their holes every single day after the tide subsided? I needed to pay attention during the rest of my vacation to figure this out.
Tides of Change
The tides brought serious problems to the crab burrows. My inner IT performance geek immediately likened the tidal damage to the problems brought about by an onslaught of user activity to a web application. As the tide built the sand pile outside of the burrow washed away. In my head I was imagining a swell of user generated workload eroding away business critical application performance. Only after enough pounding did the sand pile completely give way and the crab burrow collapsed under the stress of the ocean.
This is how I’ve seen most IT performance problems develop as well. It’s usually not an immediate collapse (system crash), but a degradation over a period of time that leads to a failure of the application.
Where Have All the Good Crabs Gone?
How did the crabs respond to all of this? At first I didn’t know where the crabs were while their burrows were under water. After some observation I realized that some had retreated to the tropical woods at the edge of the beach and that others had found shelter on, in, and around a rock wall that ran from the beach into the water. This isn’t really relevant to my story but I thought you’d be interested in finding out. The interesting part to me is what happened when the tide subsided. As the day wore on and the tide became compliant with crab low tide regulations I noticed a massive crab undertaking. Most of the crabs returned at nearly the same time and started re-digging their burrows.
I was amazed at how much sand these little crustaceans could move with a single scoop of their crabby claws. These crabs knew exactly how to fix the problem that the tide had burdened them with and within 30 minutes their burrows were completely restored.
Crabs and IT Operations
This entire process reminded me of my days in IT operations when an alert would fire, we would figure out what the problem was, and soon after problem isolation we would have service restored. That is all fine and good but here is the big problem. The crabs did the same thing every single day of my vacation. They built a burrow, the tide washed it away, and they rebuilt their burrow when the tide subsided. Day after day after day forever.
Many of us did the same thing in IT operations. After the problem was fixed and service restored we congratulated each other and enjoyed our burrows instead of taking the extra time to figure out automated detection and remediation strategies for problems we had seen before. As IT practitioners we usually have more on our plates than we can possibly handle. We live in an age of “do more with less”. The problem is that if we don’t take the extra time to automate our detection and remediation of application issues we will endlessly repeat the cycle of the crabs.
Crabs can’t use complex tools, but thankfully we can. AppDynamics application runbook automation is the cure for the crab cycle. It detects problems and remediates them automatically (you choose if you want to authorize the action or not). We’ve written a couple of other posts that describe the functionality in detail, so please read through them and get familiar with this amazing set of features…
If you’re tired of the crab cycle do yourself a favor and fix it for good. Click here to get started with your free trial of AppDynamics Pro and see how powerful, easy to use software can make your life better today.Link to this post: