Hadoop - Monitoring Extension

Use Case

 

The Hadoop monitoring extension captures metrics from Hadoop Resource Manager and/or Apache Ambari and displays them in Appdynamics Metric Browser.

 

This extension works only with the standalone machine agent.

 

 

Metrics include:

 

  • Hadoop Resource Manager
    • App status and progress: submitted, pending, running, completed, killed, and failed app count
    • Memory size, memory usage
    • Allocated containers, container count in different states
    • Node status, count of nodes in different states
    • Scheduler capacity, app and container count
  • Ambari
    • Individual host metrics including CPU, disk, memory, JVM, load, network, process, and RPC metrics
    • Service component metrics including CPU, disk, memory, JVM, load, network, process, RPC, and component-specific metrics

 

Prerequisites

 

  • Hadoop distribution that features Hadoop YARN. Distributions based on Hadoop 2.x should include YARN
    • Examples: hadoop-2.x, hadoop-0.23.x, CDH4 from Cloudera, HDP 2 from Hortonworks, and Pivotal HD from Pivotal
  • Resource Manager hostname and port

 

OR

 

  • Any release of Apache Amabri
  • Any version of Hadoop cluster installed via Ambari
  • Ambari server hostname, port, username, and password

If the cluster version is 1.x and not installed by Ambari, no metrics can be gathered at this time.

 

 

Installation 

  1. Download the attached HadoopMonitor.zip file.
  2. Unzip the HadoopMonitor.zip file into the <machineagent install dir>/monitors/ directory.
  3. Open <machineagent install dir>/monitors/HadoopMonitor/config.yml and configure the HRM and/or Ambari parameters.
  4. Restart the machine agent.
  5. In the AppDynamics Metric Browser, look for: Application Infrastructure Performance | <Tier> | Custom Metrics | Hadoop.

 

Configuration

Note : Please make sure to not use tab (\t) while editing yaml files. You may want to validate the yaml file using a yaml validator

  1. Configure the extension by editing the config.yml file in <MACHINE_AGENT_HOME>/monitors/HadoopMonitor/.

    For eg:

        # To enable or diable ResourceManager metrics, set resourceManagerMonitor to "true" or "false"
        # The Hadoop version for the cluster you want to monitor using Resource Manager.
        # Example: 1.3, 2.2, 0.23
        # RESOURCE MANAGER Web UI CONFIG: Resource Manager is only usable for Hadoop 2.x and Hadoop 0.23.x
    
        resourceManagerMonitor: true
        resourceManagerConfig:
            hadoopVersion: "2.3"
            host: "localhost"
            port: 8088
            username: ""
            password: ""
    
        # application metrics within last X number of minutes to aggregate
            aggregateAppPeriod: 15
    
    
        # To enable or diable Ambari metrics, set ambariMonitor to "true" or "false"
        # Ambari metrics are only available for clusters inistalled using Ambari, manual installs are not eligible
        # ambariConfig: Only configure if 'ambariMonitor' is set to 'true'
    
        ambariMonitor: true
        ambariConfig:
            host: "192.168.0.101"
            port: 8080
            username: "admin"
            password: "admin"
    
        numberOfThreads: 10
    
        # includeClusters: comma-separated cluster names (case sensitive) you want to gather metrics for. If empty, all clusters are reported
        # includeHosts: comma-separated host names (case sensitive) you want to gather metrics for. If empty, all hosts are reported
        # includeServices: comma-separated service names (case sensitive) you want to gather metrics for. If empty, all services are reported
        includeClusters: []
        includeHosts: []
        includeServices: []
    
        #prefix used to show up metrics in AppDynamics
        metricPathPrefix: "Custom Metrics|HadoopMonitor|"
    
  2. Configure the path to the config.yml file by editing the in the monitor.xml file in the <MACHINE_AGENT_HOME>/monitors/HadoopMonitor/ directory. Below is the sample

     <task-arguments>
         <!-- config file-->
         <argument name="config-file" is-required="true" default-value="monitors/HadoopMonitor/config.yml" />
          ....
     </task-arguments>
    

Note : By default, a Machine agent or a AppServer agent can send a fixed number of metrics to the controller. To change this limit, please follow the instructions mentioned here. For eg.

    java -Dappdynamics.agent.maxMetrics=2500 -jar machineagent.jar

 

Metrics

 

Hadoop Resource Manager:

 

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

 

ClusterMetrics

Nodes

SchedulerInfo

AggregatedApps 

Metrics under Apps are aggregated metrics from all running and recently finished apps. To specify the period for which metrics are gathered, set in properties.xml.

 

Ambari:

 

Host Metrics

To view these metrics, Ganglia and/or Nagios services should be running.

 

Metric NameDescription
CPUCPU usage
DiskDisk usage (GB)
JVMJVM log message count, memory usage, and thread count
LoadHost load in 15, 5, and 1 min averages
MemoryMemory usage (KB)
NetworkBytes in/out, packets in/out
ProcessProcess count
RPCRPC process time, queue time, sent and received bytes
RPCdetailedAdditional RPC metrics
UGIUser Group Information
part_max_usedPercent disk usage
StateHost state

 

Host States

 

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/host-resources.md#states

 

StateValueDescription
INIT0New host state.
WAITING_FOR_HOST_STATUS_UPDATES1Registration request is received from the Host but the host has not responded to its status update check.
HEALTHY2The server is receiving heartbeats regularly from the host and the state of the host is healthy.
HEARTBEAT_LOST3The server has not received a heartbeat from the host in the configured heartbeat expiry window.
UNHEALTHY4Host is in unhealthy state as reported either by the Host itself or via any other additional means ( monitoring layer ).

 

Service, Service Component and Host Component States

 

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/service-resources.md#states

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/host-component-resources.md#states

 

StateValueDescription
INIT0The initial clean state after the service is first created.
INSTALLING1In the process of installing the service.
INSTALL_FAILED2The service install failed.
INSTALLED3The service has been installed successfully but is not currently running.
STARTING4In the process of starting the service.
STARTED5The service has been installed and started.
STOPPING6In the process of stopping the service.
UNINSTALLING7In the process of uninstalling the service.
UNINSTALLED8The service has been successfully uninstalled.
WIPING_OUT9In the process of wiping out the installed service.
UPGRADING10In the process of upgrading the service.
MAINTENANCE11The service has been marked for maintenance.
UNKNOWN12The service state can not be determined.

 

Service Component

 

Metric NameDescription
CPUCPU usage
DiskDisk usage (GB)
JVMJVM log message count, memory usage, and thread count
LoadComponent load in 15, 5, and 1 min averages
MemoryMemory usage (KB)
NetworkBytes in/out, packets in/out
ProcessProcess count
RPCRPC process time, queue time, sent and received bytes
RPCdetailedAdditional RPC metrics
UGIUser Group Information
StateComponent state

 

 

Custom Dashboard

 

Hadoop 1

 

Hadoop 1 Dashboard

 

Hadoop 2

 

Hadoop 2 Dashboard

 

Contributing

 

Always feel free to fork and contribute any changes directly via GitHub.

 

Support

 

For any questions or feature request, please contact AppDynamics Center of Excellence.

 

Version:

2.2

Compatibility:

3.7

Last Update:

07/30/2015

 

Release notes:

 

1.3 -- Updated to support Hadoop 1.x installed by Ambari, as well as Hadoop 2.x. Custom dashboards updated.

2.0.1 -- Fix to work with Machine Agent 4.x

2.2 -- Added host component states, configuration moved to config.yml