In my previous blog, I wrote at length about the complexities of running a data cloud in production. This logical data set, spread across many nodes, requires a whole new set of tools and methodologies to run and maintain. Today we’ll look at one of the biggest challenges in managing a data cloud – monitoring.

Database monitoring used to be easy in the days before data clouds. Datasets were stored in a single large database, and there were hundreds of off-the-shelf products available to monitor the performance of that database. When problems occurred, one had simply to open up the monitoring tool and look at a set of graphs and metrics to diagnose the problem.

There are no off-the-shelf tools for monitoring a data cloud, however. There’s no easy way to get a comprehensive view of your entire data cloud, let alone diagnose problems and monitor performance. Database monitoring solutions simply don’t cut it in this kind of environment. So how do we monitor the performance of our data cloud? I’ll tell you what I did.

It just so happens I work at AppDynamics, one of the most powerful application monitoring tools on the market. We monitor all parts of your application including the data layer, with visibility into both Relational and NoSQL systems like Cassandra. With AppDynamics I was able to create a dashboard that gives me a single pane-of-glass view into the performance of my data cloud.

My Big Data Dashboard

This dashboard is now used in several departments at AppDynamics including Operations, QA, Performance and development teams to see how our data cloud is running. All key metrics about all of our replicas are graphed side by side on one screen. This is the dream of anyone running big data systems in production!

Of course, not all problems are system wide. More often than not you need to drill into one replica or replica set to find a problem. To do that, I simply double click on any part of my big data dashboard to focus on a single replica, change the time range, and add more metrics.

Data clouds are difficult to run, and there aren’t any database monitoring tools fit to monitor them yet. But instead of sitting around waiting for data monitoring tools to catch up with our needs, I’ve built my own Big Data Dashboard with monitoring tool designed for applications.

Of course the fun doesn’t stop here…I still need to find a way to set up alerts and do performance tuning for my data cloud. Stay tuned for more blogs in this series to see how I do it!

Big Data, database monitoring, NoSQL, sharding

As data has grown exponentially at many sites, companies have been forced to horizontally scale their data.  Some have turned to sharding of databases like Postgres or MySQL, while others have switched to newer NoSQL data systems.  There have been many debates in the last few years about SQL vs. NoSQL data management systems and which is better.  What many have failed to grasp, though, is how similar these systems are and how complex they both are to run in production in high scale.

Both of these systems represent what I call a Data Cloud. This Data Cloud is logical data set spread across many nodes.  While developers have heated debates about which system is better and how to design code around it, those in DevOps usually struggle with very similar issues because the two systems are mostly the same.  Both systems

• Run across many nodes with large amounts of data flowing between them and from/to the application
• Strain both the hardware of all nodes, and the network connecting them
• Maintain duplicate data across nodes for fault tolerance, and must have failover ability
• Must be tuned on a per node and cluster-wide bases
• Must allow for growth by adding additional nodes.

Running this Data Cloud in production presents a new set of challenges for DevOps, many of which are not well understood or addressed.  One of the main challenges is the management and monitoring of these systems, for which few (if any) tools or products exist at this time.

When systems were smaller and you ran a single Database in production, you probably had all the necessary systems in place.  With a plethora of products for Management, monitoring, visualizing data, and backups, it was not hard to be successful and meet your SLAs.

But now all this is much more complex once you move into the world of the Data Cloud.  Now you have a large number of nodes, all representing the same system and still needing to meet the same SLAs as the old simple DB from before.  Let us look at the challenges for running a production Data Cloud successfully.

Capacity Planning

Do you know how many nodes you need?  How many nodes do you put in each replica set?  How much latency and throughput do you need in your network for the nodes to communicate fast enough?  What is the ideal hardware to use for each node to balance performance with costs?

Monitoring

How do you monitor dozens, hundreds or even thousands of nodes all at once?   How do you get a unified view of your data cloud, and then drill down to the problem nodes?   Are there even any off-the-shelf monitoring tools that can help?  Your old monitoring tool won’t be very useful anymore unless you are willing to look at every node one by one to see what is going on there.

Alerting

How do you set up a common set of alerts across all nodes?  And how do you keep your alert thresholds in sync as you add nodes and remove them?   More importantly, even assuming you have alerting in place,  once staff receives critical alerts, how will they know where to find the troubled node in the massive cloud, or whether it’s a node level  issue or more global in nature?  This must be done quickly during critical outages.

Data Visualization

How does your staff view the data when it is distributed?  In case of data inaccuracy, how can they quickly identify the faulty nodes and fix up the data?

Performance Tuning

As performance degrades, how do you troubleshoot and identify the bottlenecks?  How do you find which nodes by be the cause of the problem?  How do you improve performance across all the nodes.

Data Cloud Management

How do you back up all the data while consistently tracking which nodes were backed up successfully and when?  How do you make schema changes across all the nodes in one consistent step without breaking your app? And how do you make configuration changes on various nodes or across all nodes?  And how do you track the configurations of each node and keep them consistent across your system?

By now you should see that there is a lot to think about before endeavoring to launch a production Data Cloud.  Too many companies focus all their energies on deciding which DB or NoSQL system to use and developing their apps for it.  But that might turn out to be the lesser of your challenges once you struggle to put the system into production.  Be sure you can answer all the questions I have listed above before your launch.

Boris.

Big Data, Cassandra, cloud, Data Cloud, Data Management, DevOps, NoSQL

Welcome Cassandra, let’s start with a quick introduction of who you are and what you can do.
I am a high-performance distributed scalable database.  I am really good at doing lots of operations and growing as your data needs increase, and to top that all off I run on commodity hardware which means I’m really great for running on the cloud.

Awesome, oh by the way, what beer are you drinking tonight?
Tonight I’m drinking a Shiner from good old Texas.

Excellent.  First, I heard it on the grapevine and you did tell me that you’re highly available.  Can you confirm or deny these rumors, and what does that really mean?
Highly available means that a distributed system can lose one or more of its servers and still keep the cluster as a whole up and running.  In other words, because I’m designed to run on commodity hardware, failure should be expected.  In my scenario, I can lose a server and keep on chugging along without the cluster going down.

Read the Full Post…

Apache Cassandra, appdynamics, Big Data, Cassandra, Cassandra JMX, Cassandra Monitoring, DataStax, NoSQL

Welcome Jeremy, lets start with a quick introduction of who you are and what you do at EMC.
I run marketing at EMC working for Joe Tucci, our CEO.  Been there about 18 months.

And what beer will you be drinking tonight?
That new Bud in bottles that includes Lime – why did no one think of that until now ?  I hate putting that real lime in my beer and squirting it all over my shirt.  American innovation leads the way again.

So this Cloud meets Big Data stuff, what’s all that about?
Cloud has emerged as the biggest disruptive force in IT for at least the last decade.  And maybe ever.  Complexity in IT departments is at a breaking point, so they are re-transforming their infrastructure around virtualized servers, storage, and networking, transforming their applications using frameworks like Spring and Ruby and transforming access using a myriad of consumer devices such as the iPad.   Once this transformation is complete, IT will be able to run the way God intended it to run – as an agile, efficient service.

Read the Full Post…

apm, appdynamics, Big Data, cloud, EMC, Hadoop, Jeremy Burton, NoSQL, Private Cloud, Public Cloud, VMWare