AWS StorageGateway Monitoring Extension

AWS StorageGateway Monitoring Extension

Use Case

Captures StorageGateway statistics from Amazon CloudWatch and displays them in the AppDynamics Metric Browser.

Note : By default, the Machine agent can only send a fixed number of metrics to the controller. This extension potentially reports thousands of metrics, so to change this limit, please follow the instructions mentioned here.

Prerequisites

  1. Please give the following permissions to the account being used to with the extension.
    cloudwatch:ListMetrics
    cloudwatch:GetMetricStatistics
    
  2. In order to use this extension, you do need a Standalone JAVA Machine Agent or SIM Agent. For more details on downloading these products, please visit here.
  3. The extension needs to be able to connect to AWS Cloudwatch in order to collect and send metrics. To do this, you will have to either establish a remote connection in between the extension and the product using access key and secret key, or have an agent running on EC2 instance, which you can use with instance profile.

Agent Compatibility:

Note: This extension is compatible with Machine Agent version 4.5.13 or later.

  1. If you are seeing warning messages while starting the Machine Agent, update the http-client and http-core JARs in {MACHINE_AGENT_HOME}/monitorsLibs to httpclient-4.5.9 and httpcore-4.4.12 to make this warning go away.

  2. To make this extension work on Machine Agent < 4.5.13, the http-client and http-core JARs in {MACHINE_AGENT_HOME}/monitorsLibs need to be updated to httpclient-4.5.9 and httpcore-4.4.12.

Installation

  1. Run 'mvn clean install' from aws-storagegateway-monitoring-extension
  2. Copy and unzip AWSStorageGatewayMonitor-<version>.zip from 'target' directory into <machine_agent_dir>/monitors/
  3. Edit config.yml file in AWSStorageGatewayMonitor/conf and provide the required configuration (see Configuration section)
  4. Restart the Machine Agent.

Configuration

config.yml

Note: Please avoid using tab (\t) when editing yml files. You may want to validate the yml file using a yaml validator.

SectionFieldsDescriptionExample
metricPrefixThe path prefix for viewing metrics in the metric browser."Custom Metrics|Amazon StorageGateway|",
"Server|Component:<TIER_ID>|Custom Metrics|Amazon StorageGateway|"
accountsFields under this section can be repeated for multiple accounts config
awsAccessKeyAWS Access Key
awsSecretKeyAWS Secret Key
displayAccountNameDisplay name used in metric path"MyAWSStorageGateway"
regionsRegions where StorageGateway is registered
credentialsDecryptionConfig---------------
enableDecryptionIf set to "true", then all aws credentials provided (access key and secret key) will be decrypted - see AWS Credentials Encryption section
encryptionKeyThe key used when encrypting the credentials
proxyConfig---------------
hostThe proxy host (must also specify port)
portThe proxy port (must also specify host)
usernameThe proxy username (optional)
passwordThe proxy password (optional)
dimensions---------------
nameDimension namerefer: AWS Storage Gateway for available Dimensions
displayNamedisplayName to be appended in metricPath
valuesaccepts an array of dimension valuesput ".*" to match all
metricsConfig---------------
includeMetricsFields under this section can be repeated for multiple metric types overrideConfigure it with the list of StorageGateway metrics which should be monitored
nameThe metric name"AvailabilityNotifications"
aliasAlias for the metrric name yo be used in metricPath
---------------
metricsTimeRange
startTimeInMinsBeforeNowThe no of mins to deduct from current time for start time of query5
endTimeInMinsBeforeNowThe no of mins to deduct from current time for end time of query.
Note, this must be less than startTimeInMinsBeforeNow
0
---------------
maxErrorRetrySizeThe max number of retry attempts for failed retryable requests1
concurrencyConfig
noOfAccountThreadsThe no of threads to process multiple accounts concurrently3
noOfRegionThreadsPerAccountThe no of threads to process multiple regions per account concurrently3
noOfMetricThreadsPerRegionThe no of threads to process multiple metrics per region concurrently3
---------------
cloudWatchMonitoringBasicMonitoring Types. For more details: referAllowed Values
"Basic",
"Detailed"
---------------

Below is an example config for monitoring multiple accounts and regions:


metricPrefix: "Custom Metrics|Amazon StorageGateway|"
accounts:
  - awsAccessKey: "XXXXXXXX1"
    awsSecretKey: "XXXXXXXXXX1"
    displayAccountName: "TestAccount_1"
    regions: ["us-east-1","us-west-1","us-west-2"]

  - awsAccessKey: "XXXXXXXX2"
    awsSecretKey: "XXXXXXXXXX2"
    displayAccountName: "TestAccount_2"
    regions: ["eu-central-1","eu-west-1"]

credentialsDecryptionConfig:
    enableDecryption: "false"
    encryptionKey:

proxyConfig:
    host:
    port:
    username:
    password:
    
dimensions:
  - name: "GatewayId"
    displayName: "Gateway Id"
    values: [".*"]
  - name: "VolumeId"
    displayName: "Volume Id"
    values: [".*"]

metricsConfig:
    includeMetrics:
       - name: "AvailabilityNotifications"
         alias: "Availability Notifications"
         statType: "sum"
         aggregationType: "AVERAGE"
         timeRollUpType: "AVERAGE"
         clusterRollUpType: "INDIVIDUAL"
         delta: false
         multiplier: 1
       - name: "CacheHitPercent"
         alias: "CacheHit Percent"
         statType: "sum"
         aggregationType: "AVERAGE"
         timeRollUpType: "AVERAGE"
         clusterRollUpType: "INDIVIDUAL"
         delta: false
         multiplier: 1
       - name: "CacheUsed"
         alias: "Cache Used"
         statType: "sum"
         aggregationType: "AVERAGE"
         timeRollUpType: "AVERAGE"
         clusterRollUpType: "INDIVIDUAL"
         delta: false
         multiplier: 1
       - name: "IndexEvictions"
         alias: "Index Evictions"
         statType: "sum"
         aggregationType: "AVERAGE"
         timeRollUpType: "AVERAGE"
         clusterRollUpType: "INDIVIDUAL"
         delta: false
         multiplier: 1

    metricsTimeRange:
      startTimeInMinsBeforeNow: 5
      endTimeInMinsBeforeNow: 0
      
    getMetricStatisticsRateLimit: 400
    maxErrorRetrySize: 0

concurrencyConfig:
  noOfAccountThreads: 3
  noOfRegionThreadsPerAccount: 3
  noOfMetricThreadsPerRegion: 3
  #Thread timeout in seconds
  threadTimeOut: 30

cloudWatchMonitoring: "Basic"

AWS Credentials Encryption

To set an encrypted awsAccessKey and awsSecretKey in config.yml, follow the steps below:

  1. Download the util jar to encrypt the AWS Credentials from here.

  2. Run command:

    	java -cp appd-exts-commons-1.1.2.jar com.appdynamics.extensions.crypto.Encryptor EncryptionKey CredentialToEncrypt
    
    	For example:
    	java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsAccessKey
    
    	java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor test myAwsSecretKey
    
  3. Set the decryptionKey field in config.yml with the encryption key used, as well as the resulting encrypted awsAccessKey and awsSecretKey in their respective fields.

Please visit this page to get detailed instructions on password encryption. The steps in this document will guide you through the whole process.

Metrics

Typical metric path: Application Infrastructure Performance|<Tier>|Custom Metrics|Amazon StorageGateway|<Account Name>|<Region>|Gateway Id|<Gateway Id>|Gateway Name|<Gateway Name> followed by the metrics defined in the link below:

Extensions Workbench

Workbench is an inbuilt feature provided with each extension in order to assist you to fine tune the extension setup before you actually deploy it on the controller. Please review the following document on How to use the Extensions WorkBench

Troubleshooting

Please follow the steps listed in this troubleshooting-document in order to troubleshoot your issue. These are a set of common issues that customers might have faced during the installation of the extension. If these don't solve your issue, please follow the last step on the troubleshooting-document to contact the support team.

Support Tickets

If after going through the Troubleshooting Document you have not been able to get your extension working, please file a ticket and add the following information.

Please provide the following in order for us to assist you better.

  1. Stop the running machine agent.
  2. Delete all existing logs under <MachineAgent>/logs.
  3. Please enable debug logging by editing the file <MachineAgent>/conf/logging/log4j.xml. Change the level value of the following <logger> elements to debug.
    <logger name="com.singularity">
    <logger name="com.appdynamics">
    
  4. Start the machine agent and please let it run for 10 mins. Then zip and upload all the logs in the directory <MachineAgent>/logs/*.
  5. Attach the zipped <MachineAgent>/conf/* directory here.
  6. Attach the zipped <MachineAgent>/monitors/ExtensionFolderYouAreHavingIssuesWith directory here.

For any support related questions, you can also contact help@appdynamics.com.

Contributing

Always feel free to fork and contribute any changes directly here on GitHub.

Version

NameVersion
Extension Version2.0.0
Controller Compatibility4.4 or Later
Agent Compatibility4.5.13 or later
Last Update19 May, 2020
List of changes to this extension can be found here