Hadoop - Monitoring Extension

Use Case

The Hadoop monitoring extension captures metrics from Hadoop Resource Manager and/or Apache Ambari and displays them in Appdynamics Metric Browser.

This extension works only with the standalone machine agent.

Metrics include:

• Hadoop Resource Manager
  • App status and progress: submitted, pending, running, completed, killed, and failed app count
  • Memory size, memory usage
  • Allocated containers, container count in different states
  • Node status, count of nodes in different states
  • Scheduler capacity, app and container count
• Ambari
  • Individual host metrics including CPU, disk, memory, JVM, load, network, process, and RPC metrics
  • Service component metrics including CPU, disk, memory, JVM, load, network, process, RPC, and component-specific metrics

Prerequisites

• Hadoop distribution that features Hadoop YARN. Distributions based on Hadoop 2.x should include YARN
  • Examples: hadoop-2.x, hadoop-0.23.x, CDH4 from Cloudera, HDP 2 from Hortonworks, and Pivotal HD from Pivotal
• Resource Manager hostname and port

OR

• Any release of Apache Amabri
• Any version of Hadoop cluster installed via Ambari
• Ambari server hostname, port, username, and password

If the cluster version is 1.x and not installed by Ambari, no metrics can be gathered at this time.

Installation 

1. Download the attached HadoopMonitor.zip file.
2. Unzip the HadoopMonitor.zip file into the <machineagent install dir>/monitors/ directory.
3. Open <machineagent install dir>/monitors/HadoopMonitor/config.yml and configure the HRM and/or Ambari parameters.
4. Restart the machine agent.
5. In the AppDynamics Metric Browser, look for: Application Infrastructure Performance | <Tier> | Custom Metrics | Hadoop.

Configuration

Note : Please make sure to not use tab (\t) while editing yaml files. You may want to validate the yaml file using a yaml validator

1. Configure the extension by editing the config.yml file in <MACHINE_AGENT_HOME>/monitors/HadoopMonitor/.

   For eg:

       # To enable or diable ResourceManager metrics, set resourceManagerMonitor to "true" or "false"
       # The Hadoop version for the cluster you want to monitor using Resource Manager.
       # Example: 1.3, 2.2, 0.23
       # RESOURCE MANAGER Web UI CONFIG: Resource Manager is only usable for Hadoop 2.x and Hadoop 0.23.x
   
       resourceManagerMonitor: true
       resourceManagerConfig:
           hadoopVersion: "2.3"
           host: "localhost"
           port: 8088
           username: ""
           password: ""
   
       # application metrics within last X number of minutes to aggregate
           aggregateAppPeriod: 15
   
   
       # To enable or diable Ambari metrics, set ambariMonitor to "true" or "false"
       # Ambari metrics are only available for clusters inistalled using Ambari, manual installs are not eligible
       # ambariConfig: Only configure if 'ambariMonitor' is set to 'true'
   
       ambariMonitor: true
       ambariConfig:
           host: "192.168.0.101"
           port: 8080
           username: "admin"
           password: "admin"
   
       numberOfThreads: 10
   
       # includeClusters: comma-separated cluster names (case sensitive) you want to gather metrics for. If empty, all clusters are reported
       # includeHosts: comma-separated host names (case sensitive) you want to gather metrics for. If empty, all hosts are reported
       # includeServices: comma-separated service names (case sensitive) you want to gather metrics for. If empty, all services are reported
       includeClusters: []
       includeHosts: []
       includeServices: []
   
       #prefix used to show up metrics in AppDynamics
       metricPathPrefix: "Custom Metrics|HadoopMonitor|"
   

2. Configure the path to the config.yml file by editing the in the monitor.xml file in the <MACHINE_AGENT_HOME>/monitors/HadoopMonitor/ directory. Below is the sample

    <task-arguments>
        <!-- config file-->
        <argument name="config-file" is-required="true" default-value="monitors/HadoopMonitor/config.yml" />
         ....
    </task-arguments>
   

Note : By default, a Machine agent or a AppServer agent can send a fixed number of metrics to the controller. To change this limit, please follow the instructions mentioned here. For eg.

    java -Dappdynamics.agent.maxMetrics=2500 -jar machineagent.jar

Metrics

Hadoop Resource Manager:

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

ClusterMetrics

Nodes

SchedulerInfo

AggregatedApps 

Metrics under Apps are aggregated metrics from all running and recently finished apps. To specify the period for which metrics are gathered, set in properties.xml.

Ambari:

Host Metrics

To view these metrics, Ganglia and/or Nagios services should be running.

Host States

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/host-resources.md#states

Service, Service Component and Host Component States

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/service-resources.md#states

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/host-component-resources.md#states

Service Component

Custom Dashboard

Hadoop 1

Hadoop 2

Contributing

Always feel free to fork and contribute any changes directly via GitHub.

Support

For any questions or feature request, please contact AppDynamics Center of Excellence.

Release notes:

1.3 -- Updated to support Hadoop 1.x installed by Ambari, as well as Hadoop 2.x. Custom dashboards updated.

2.0.1 -- Fix to work with Machine Agent 4.x

2.2 -- Added host component states, configuration moved to config.yml