AWS ElasticMapReduce Monitoring Extension

AWS ElasticMapReduce Monitoring Extension

Use Case

Captures ElasticMapReduce statistics from Amazon CloudWatch and displays them in the AppDynamics Metric Browser.

This extension works only with the standalone machine agent.

Note: By default, the Machine agent can only send a fixed number of metrics to the controller. This extension can potentially report thousands of metrics, so to change this limit, please follow the instructions mentioned here.

Installation

1. Download and unzip AWSElasticMapReduceMonitor-<version>.zip into <machine_agent_dir>/monitors/
2. Edit config.yaml file in AWSElasticMapReduceMonitor/conf and provide the required configuration (see Configuration section)
3. Restart the Machine Agent.

Configuration

config.yaml

Note: Please avoid using tab (\t) when editing yaml files. You may want to validate the yaml file using a yaml validator.

Below is an example config for monitoring multiple accounts and regions:

accounts:
  - awsAccessKey: "XXXXXXXX1"
    awsSecretKey: "XXXXXXXXXX1"
    displayAccountName: "TestAccount_1"
    regions: ["us-east-1","us-west-1","us-west-2"]

  - awsAccessKey: "XXXXXXXX2"
    awsSecretKey: "XXXXXXXXXX2"
    displayAccountName: "TestAccount_2"
    regions: ["eu-central-1","eu-west-1"]

credentialsDecryptionConfig:
    enableDecryption: "false"
    decryptionKey:

proxyConfig:
    host: 
    port:
    username:
    password:    

metricsConfig:
    metricTypes:
      - metricName: "GroupInServiceInstances"
        statType: "max"

      - metricName: "GroupMinSize"
        statType: "sum"        

    excludeMetrics: ["GroupMaxSize", "GroupTotal.*"]

    metricsTimeRange:
      startTimeInMinsBeforeNow: 5
      endTimeInMinsBeforeNow: 0

    maxErrorRetrySize: 0

concurrencyConfig:
  noOfAccountThreads: 3
  noOfRegionThreadsPerAccount: 3
  noOfMetricThreadsPerRegion: 3

metricPrefix: "Custom Metrics|Amazon ElasticMapReduce|"

AWS Credentials Encryption

To set an encrypted awsAccessKey and awsSecretKey in config.yaml, follow the steps below:

1. Download the util jar to encrypt the AWS Credentials from here.

2. Run command:

   java -cp appd-exts-commons-1.1.2.jar com.appdynamics.extensions.crypto.Encryptor EncryptionKey CredentialToEncrypt
   
   For example: 
   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsAccessKey
   
   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsSecretKey
   

3. Set the decryptionKey field in config.yaml with the encryption key used, as well as the resulting encrypted awsAccessKey and awsSecretKey in their respective fields.

Metrics

Typical metric path: Application Infrastructure Performance|<Tier>|Custom Metrics|Amazon ElasticMapReduce|<Account Name>|<Region>|JobFlow Id|<JobFlow Id> followed by the metrics defined in the link below:

• ElasticMapReduce Metrics

Contributing

Always feel free to fork and contribute any changes directly via GitHub.

Support

For any questions or feature request, please contact AppDynamics Center of Excellence.