AppDynamics Monitoring Extension for use with Docker

Introduction

Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.Docker Monitoring extension gathers metrics from the Docker Remote API, either using Unix Socket or TCP.

Prerequisites

1. Please go through the 1st section of the doc and review the configuration. https://docs.docker.com/reference/api/docker_remote_api/
2. The Stats API GET /containers/(id)/stats is available only from Docker API version 1.17 (Docker version ~1.8) onwards. If you are using an older version, the CPU Stats, Memory Stats and Network Stats will not be available.
3. TCP Socket: The docker daemon should be bound to the tcp socket. Please refer to https://docs.docker.com/engine/reference/commandline/dockerd/. This is the command to bind the docker daemon to both TCP Socket and Unix Socket sudo /path/to/docker daemon -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock &
4. Unix Socket: The extension will be able to fetch the data over the Unix Sockets if the CURL v 7.40+ is installed. Fetching data through the Unix Socket with older versions of CURL is not supported.
5. Unix Socket: To use this mode to collect the data, the machine agent should be run as the root user. If this is not possible, then the current user should have password-less sudo access or he should have access to the docker socket

Installation

1. Please start the Machine Agent before installing the extension and make sure that it reports data. Verify that the machine-agent status is UP and it is reporting Hardware Metrics
2. Download and unzip the DockerMonitor.zip to the <MachineAgent_Dir>/monitors directory
3. Edit the file config.yml located at <MachineAgent_Dir>/monitors/DockerMonitor and update the following details. Comment out properties which are not used
   metricPrefix: To report the metrics only to a given Tier, use the second one instead. The TIER_ID can be found from the REST API

   metricPrefix: Custom Metrics|Docker
   #metricPrefix: Server|Component:<TIER_ID>|Custom Metrics|Docker

   unixSocket

   #Needs curl v7.40+ to enable unix sockets
   unixSocket:
       commandFile: monitors/DockerMonitor/socket-command.sh

   tcpSockets: Multiple TCP sockets can be added here. Each one should have a base URL and a unique display name. The metrics will be reported under this name.

   #TCP sockets should be bound to docker demon. Make sure that the port is open, 2375 in this case
   tcpSockets:
       - baseUrl: http://127.0.0.1:2375
         name: Server1

4. Custom Dashboard: Update the following properties in the customDashboard section
   
   1. username [Required] A user that can login to controller ui and upload dashboard
   2. password[Required*] Clear text password for the user to upload the dashboard. Optionally use the passwordEncrypted and encryptionKey
   3. account [Required] the AppDynamics tenant account name
   4. passwordEncrypted[Optional*] See the section Password Encryption Support
   5. encryptionKey[Optional*] See the section Password Encryption Support
   6. applicationName [Required]
   7. tierName [Required]
   
   
5. Please review the contents of the file at the location <MachineAgent>/monitors/DockerMonitor/socket-command.sh


6. Add executable permissions

   chmod +x MachineAgent/monitors/DockerMonitor/socket-command.sh

Custom Dashboard

The extension will generate and upload the following custom dashboard to the controller. This feature requires the version 4x of Machine Agent and Controller. Please look at the dashboard section in config.yml for configuration.

Please make sure that the customDashboard section of config.yml is configured correctly.

Over time, you might need to update contents of the dashboard.To create a new dashboard, delete(or rename) the existing dashboard and let extension upload a new one. See troubleshooting steps 4 and 5.

Metrics

The metrics will be reported under the tree Application Infrastructure Performance|$TIER|Custom Metrics|Docker

Please refer to this screenshot to view the complete list of metrics reported by the Docker Extension.

Credentials Encryption

Please visit this page to get detailed instructions on password encryption. The steps in this document will guide you through the whole process.

Extensions Workbench

Workbench is an inbuilt feature provided with each extension in order to assist you to fine tune the extension setup before you actually deploy it on the controller. Please review the following document on How to use the Extensions WorkBench

Troubleshooting

1. Verify Machine Agent Data:Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics
2. Unix Socket: This needs curl v7.40+. To troubleshoot unix socket, please try to run this command and see if it returns a valid JSON data in addition to the http headers. If needed, change the API from /containers/id/stats to whichever one you want to test.

   sudo curl -s -S -i --unix-socket /var/run/docker.sock  http:/containers/$CONTAINER_ID/stats

   Please update the $CONTAINER_ID. If this API doesn't return any JSON data in the response in addition to http headers, then you will have to enable TCP sockets. Please refer to Prerequisites#3 on how to enable TCP Sockets. Then modify the config.yml and uncomment the tcpSockets section accordingly. Comment out the unixSocket section in config.yml
3. TCP Socket: To troubleshoot TCP, make sure that the following commands returns a valid JSON output

   wget http://127.0.0.1:2375/containers/json -O - -q
   wget http://127.0.0.1:2375/containers/$CONTAINER_ID/stats -O - -q
   

   If you are executing the command from different machine other than the Docker Host Machine, please update the host part of the above url 127.0.0.1 to the IP of the Docker Host Machine. Also please update the port value 2375 if needed. Please refer to Prerequisites#3 section for more details. Please update $CONTAINER_ID accordingly
4. Metric Limit: Please start the machine agent with the argument -Dappdynamics.agent.maxMetrics=1000, if there is a metric limit reached error in the logs
5. Dashboard Upload: By default, the dashboard upload works only with the SSL in SAAS. If the machine agent connects to SAAS servers on port 80, then you need to set these properties in the customDashboard section of config.yml.. The properties are controllerPort: 443 and controllerSslEnabled: true. This is applicable only to AppDynamics SAAS Environment.
6. Dashboard Upload: If there are any issues in uploading the dashboard, update the value of uploadDashboard to false in config.yml. This will create an xml in the logs directory. Please upload it manually through the controller ui. To debug any issues, attach it to the support ticket.
7. Recreate Dashboard: If the existing dashboard is deleted from the controller ui, then the extension will create a new dashboard.
8. Please follow the steps listed in this troubleshooting-document in order to troubleshoot your issue. These are a set of common issues that customers might have faced during the installation of the extension. If these don't solve your issue, please follow the last step on the troubleshooting-document to contact the support team.

Support Tickets

If after going through the Troubleshooting Document you have not been able to get your extension working, please file a ticket and add the following information.

Please provide the following in order for us to assist you better.

1. Stop the running machine agent .
2. Delete all existing logs under <MachineAgent>/logs.
3. Please enable debug logging by editing the file <MachineAgent>/conf/logging/log4j.xml. Change the level value of the following <logger> elements to debug.
   • <logger name="com.singularity">
   • <logger name="com.appdynamics">
4. Start the machine agent and please let it run for 10 mins. Then zip and upload all the logs in the directory <MachineAgent>/logs/*.
5. Attach the zipped <MachineAgent>/conf/* directory here.
6. Attach the zipped <MachineAgent>/monitors/ExtensionFolderYouAreHavingIssuesWith directory here .

For any support related questions, you can also contact help@appdynamics.com.

Known Issues

• The extension doesn't work with the newer versions Docker Unix Socket. Please enable the TCP socket. For details, please refer to the Troubleshooting #1

Compatibility

Releases Notes

• 1.0.20: Server|Component:<TIER_NAME> is no longer supported by machine agent. Have to use Server|Component:<TIER_ID>
• 1.0.24: Defaulted the Null values to zero
• 1.0.25: External Library Update
• 1.0.26: Added CURL command to socket-command.sh
• 1.0.28: Switched to curl by default on unix sockets
• 1.0.29: Container Naming customization
• 1.0.30: Updated the dashboard upload API
• 1.0.31: Added License files
• 1.0.32: Fix for custom dashboard error