1. Use Case

The Hadoop monitoring extension captures metrics from Hadoop Resource Manager and/or Apache Ambari and displays them in Appdynamics Metric Browser.

This extension works only with the standalone machine agent.

2. Prerequisites

2.1 Ambari Metrics

To capture the Ambari Metrics, the Ambari Metrics Collector and Metrics Collector must be enabled.

2.2 Resource Manager Metrics

Extension collects the metric from the Resource Manager Rest API. The installation should include Hadoop Yarn for this to work.

3. Installation

1. Download and Unzip the attachment into MachineAgent/monitors directory

4. Configuration

4.1 Resource Manager Monitor

Resource Manger Monitor can be used if you have Hadoop Yarn. Please refer to Resource Manager Rest API for mor details

1. Edit the file HadoopMonitor/config.yml. Locate the section resourceManagerMonitor:. The following instructions are applicable to this section only
2. enabled: This should be true if you want to collect metrics through Ambari
3. metricPrefix: The metrics will be reported under this tree in the metric browser. You don't have to change this
4. servers: The sub elements of this section defines the details of the Resource Manager Server. If you are monitoring multiple Resource Managers, then you can add the details one by one.
   1. uri: The uri of the Resource Manager. You only need to change the scheme, host and port in the uri
   2. username: The username to login to Resource Manager Host. Uncomment if you have authentication enabled
   3. password: The password to login to Resource Manager Host. Uncomment if you have authentication enabled
   4. name: Required only if there is more than one server. Add any name to identify a server, preferred name is the host name of the server.
5. applications: The extension will monitor the state of applications in the given monitoringTimePeriod. It will report the number of running, finished, failed applications. You can add the applicationType and a regex that matches the application name to monitor.
   1. type: The application Type eg. MAPREDUCE
   2. names: Includes all [.*] by default. Can customize, for example to add all application names which start with app1 and app3 use ["app1.*","app3.*"]
6. monitoringTimePeriod: (In Minutes) This is in reference with the above applications element. It will monitor the state of the applications in the given time period.

 

4.2 Ambari Monitor

Ambari Monitor can be used if you have installed the hadoop cluster using Ambari.

1. Edit the file HadoopMonitor/config.yml. Locate the section ambariMonitor:. The following instructions are applicable to this section only
2. enabled: This should be true if you want to collect metrics through Ambari
3. metricPrefix: The metrics will be reported under this tree in the metric browser. You don't have to change this
4. servers: The sub elements of this section defines the details of the Ambari Host. If you are monitoring multiple Ambari Installations, then you can add the details one by one.
   1. uri: The uri of the Ambari Host. You only need to change the scheme, host and port in the uri
   2. username: The username to login to Ambari Host
   3. password: The password to login to Ambari Host.
   4. name: Uncomment only if there is more than one server. Add any name to identify a server, preferred name is the host name of the server.
5. filters: This section defines filters i.e meaning what to include. It supports multiple regular expressions, separated bt comma.

5. Password Encryption Support

To avoid setting the clear text password in the config.yml, please follow the process to encrypt the password and set the encrypted password and the key in the config.yml

1. Download the util jar to encrypt the password from here
2. Encrypt password from the commandline

   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor myKey myPassword

3. These values should be used in the passwordEncrypted and encryptionKey fields in config.yml under the section servers: instead of the element password

6. Troubleshooting

1. Verify Machine Agent Data: Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics
2. config.yml: Make sure that the file is valid syntactically. Validate the file here
3. ResourceManager REST API Please make sure that the ResourceManager REST API is available and accessible. Try this URL from the Browser http://host:8088/ws/v1/cluster/metrics, after replacing the host and port.
4. Ambari REST API Please make sure that the Ambari REST API is available and accessible. Try this URL from the Browser http://host:8080/api/v1/clusters?fields=Clusters,hosts,services, after replacing the host and port.
5. Metric Limit: Please start the machine agent with the argument -Dappdynamics.agent.maxMetrics=5000 if there is a metric limit reached error in the logs. If you dont see the expected metrics, this could be the cause.
6. Check Logs: There could be some obvious errors in the machine agent logs. Please take a look.
7. Collect Debug Logs: Edit the file, <MachineAgent>/conf/logging/log4j.xml and update the level of the appender com.appdynamics to debug Let it run for 5-10 minutes and attach the logs to a support ticket

7. Workbench

Workbench is a feature that lets you preview the metrics before registering it with the controller. This is useful if you want to fine tune the configurations. Workbench is embedded into the extension jar.

To use the workbench

1. Follow all the installation steps
2. Start the workbench with the command

   java -jar /path/to/MachineAgent/monitors/HadoopMonitor/hadoop-monitoring-extension.jar

   This starts an http server at http://host:9090/. This can be accessed from the browser.
3. If the server is not accessible from outside/browser, you can use the following end points to see the list of registered metrics and errors.

   #Get the stats
   curl http://localhost:9090/api/stats
   #Get the registered metrics
   curl http://localhost:9090/api/metric-paths
   

4. You can make the changes to config.yml and validate it from the browser or the API
5. Once the configuration is complete, you can kill the workbench and start the Machine Agent

8. Metrics

The metrics will be reported under the tree Application Infrastructure Performance|$TIER|Custom Metrics|Hadoop Monitor|

8.1 Ambari Metrics List

Please refer to this screenshot to view the complete list of metrics reported by the Ambari Monitor.

8.2 ResourceManager Metrics List

Please refer to this screenshot to view the complete list of metrics reported by the ResourceManager Monitor.

9. Contributing

Always feel free to fork and contribute any changes directly via GitHub.

10. Support

For any questions or feature request, please contact AppDynamics Help.

11. Compatibility

12. Releases

• 1.3.0: Updated to support Hadoop 1.x installed by Ambari, as well as Hadoop 2.x
• 2.0.1: Fix to work with Machine Agent 4.x
• 1.0.4: Fixed self-signed certificates issue
• 2.2.0: Added host component states, configuration moved to config.yml
• 2.5.0: Made the list of metrics configurable