AWS ElasticMapReduce Monitoring Extension

AWS ElasticMapReduce Monitoring Extension

Use Case

Captures ElasticMapReduce statistics from Amazon CloudWatch and displays them in the AppDynamics Metric Browser.

This extension works only with the standalone machine agent.

Note: By default, the Machine agent can only send a fixed number of metrics to the controller. This extension can potentially report thousands of metrics, so to change this limit, please follow the instructions mentioned here.

Prerequisite

1. Before the extension is installed, the prerequisites mentioned here need to be met. Please do not proceed with the extension installation if the specified prerequisites are not met.

Installation

1. Download and unzip AWSElasticMapReduceMonitor-<version>.zip into <machine_agent_dir>/monitors/
2. Edit the file config.yaml located at <MachineAgent_Dir>/monitors/AWSElasticMapReduceMonitor/conf The metricPrefix of the extension has to be configured as specified here. Please make sure that the right metricPrefix is chosen based on your machine agent deployment, otherwise this could lead to metrics not being visible in the controller.
3. Restart the Machine Agent.

Configuration

config.yaml

Note: Please avoid using tab (\t) when editing yaml files. You may want to validate the yaml file using a yaml validator.

Below is an example config for monitoring multiple accounts and regions:

accounts:
  - awsAccessKey: "XXXXXXXX1"
    awsSecretKey: "XXXXXXXXXX1"
    displayAccountName: "TestAccount_1"
    regions: ["us-east-1","us-west-1","us-west-2"]

  - awsAccessKey: "XXXXXXXX2"
    awsSecretKey: "XXXXXXXXXX2"
    displayAccountName: "TestAccount_2"
    regions: ["eu-central-1","eu-west-1"]

credentialsDecryptionConfig:
    enableDecryption: "false"
    decryptionKey:

proxyConfig:
    host: 
    port:
    username:
    password:    

metricsConfig:
    metricTypes:
      - metricName: "GroupInServiceInstances"
        statType: "max"

      - metricName: "GroupMinSize"
        statType: "sum"        

    excludeMetrics: ["GroupMaxSize", "GroupTotal.*"]

    metricsTimeRange:
      startTimeInMinsBeforeNow: 5
      endTimeInMinsBeforeNow: 0

    maxErrorRetrySize: 0

concurrencyConfig:
  noOfAccountThreads: 3
  noOfRegionThreadsPerAccount: 3
  noOfMetricThreadsPerRegion: 3

metricPrefix: "Custom Metrics|Amazon ElasticMapReduce|"

AWS Credentials Encryption

To set an encrypted awsAccessKey and awsSecretKey in config.yaml, follow the steps below:

1. Download the util jar to encrypt the AWS Credentials from here.

2. Run command:

   java -cp appd-exts-commons-1.1.2.jar com.appdynamics.extensions.crypto.Encryptor EncryptionKey CredentialToEncrypt
   
   For example: 
   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsAccessKey
   
   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsSecretKey
   

3. Set the decryptionKey field in config.yaml with the encryption key used, as well as the resulting encrypted awsAccessKey and awsSecretKey in their respective fields.

Metrics

Typical metric path: Application Infrastructure Performance|<Tier>|Custom Metrics|Amazon ElasticMapReduce|<Account Name>|<Region>|JobFlow Id|<JobFlow Id> followed by the metrics defined in the link below:

• ElasticMapReduce Metrics

Troubleshooting

1. Please look at the troubleshooting document and make sure that everything is followed correctly.

Contributing

Always feel free to fork and contribute any changes directly via GitHub.

Support

For any questions or feature request, please contact AppDynamics Center of Excellence.