Docker Monitoring Extension

Introduction

Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.Docker Monitoring extension gathers metrics from the Docker Remote API, either using Unix Socket or TCP.

Prerequisites

1. Please go through the 1st section of the doc and review the configuration. https://docs.docker.com/reference/api/docker_remote_api/
2. The Stats API GET /containers/(id)/stats is available only from Docker version 1.17 onwards. If you are using an older version, the CPU Stats, Memory Stats and Network Stats will not be available.
3. TCP Socket: The docker daemon should be bound to the tcp socket. Please refer to https://docs.docker.com/articles/basics/#bind-docker-to-another-hostport-or-a-unix-socket. This is the command to bind the docker daemon to both TCP Socket and Unix Socket sudo /path/to/docker daemon -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock &
4. Unix Socket: There are some known issues while using UnixSocket to fetch the data. Please refer to Troubleshooting / Known Issues Section
5. Unix Socket: To use this mode to collect the data, the machine agent should be run as the root user. If this is not possible, then the current user should have password-less sudo access or he should have access to the docker socket
6. Unix Socket: netcat (nc) is required to fetch the data from socket. Please install it.

Installation

1. Please start the Machine Agent before installing the extension and make sure that it reports data. Verify that the machine-agent status is UP and it is reporting Hardware Metrics
2. Download and unzip the DockerMonitor.zip to the <MachineAgent_Dir>/monitors directory
3. Edit the file config.yml located at <MachineAgent_Dir>/monitors/DockerMonitor and update the following details. Comment out properties which are not used
   metricPrefix: To report the metrics only to a given Tier, use the second one instead. The TIER_ID can be found from the REST API

   metricPrefix: Custom Metrics|Docker
   #metricPrefix: Server|Component:<TIER_ID>|Custom Metrics|Docker

   unixSocket

   unixSocket:
       commandFile: monitors/DockerMonitor/socket-command.sh

   tcpSockets: Multiple TCP sockets can be added here. Each one should have a base URL and a unique display name. The metrics will be reported under this name.

   tcpSockets:
       - baseUrl: http://127.0.0.1:2375
         name: Server1

4. Custom Dashboard: Update the following properties in the customDashboard section
   
   1. username [Required] A user that can login to controller ui and upload dashboard
   2. password[Required*] Clear text password for the user to upload the dashboard. Optionally use the passwordEncrypted and encryptionKey
   3. passwordEncrypted[Optional*] See the section Password Encryption Support
   4. encryptionKey[Optional*] See the section Password Encryption Support
   5. applicationName [Required]
   6. tierName [Required]
   
   
5. Please review the contents of the file at the location <MachineAgent>/monitors/DockerMonitor/socket-command.sh


6. Add executable permissions

   chmod +x MachineAgent/monitors/DockerMonitor/socket-command.sh

Custom Dashboard

The extension will generate and upload the following custom dashboard to the controller. This feature requires the version 4x of Machine Agent and Controller. Please look at the dashboard section in config.yml for configuration.

Please make sure that the customDashboard section of config.yml is configured correctly.

Over time, you might need to update contents of the dashboard.To create a new dashboard, delete(or rename) the existing dashboard and let extension upload a new one. See troubleshooting steps 4 and 5.

Password Encryption Support

To avoid setting the clear text password in the config.yml, please follow the process to encrypt the password and set the encrypted password and the key in the config.yml

1. Download the util jar to encrypt the password from  here
2. Encrypt password from the commandline

   java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor myKey myPassword

Troubleshooting

1. Verify Machine Agent Data:Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics
2. Unix Socket: To troubleshoot unix socket, please try to run this command and see if it returns a valid JSON data in addition to the http headers. If needed, change the API from /containers/id/stats to whichever one you want to test.

   echo -e "GET /containers/$CONTAINER_ID/stats HTTP/1.0\r\n" | sudo nc -U /var/run/docker.sock

   Please update the $CONTAINER_ID. If this API doesn't return any JSON data in the response in addition to http headers, then you will have to enable TCP sockets. Please refer to Prerequisites#3 on how to enable TCP Sockets. Then modify the config.yml and uncomment the tcpSockets section accorgingly. Comment out the unixSocket section in config.yml

   For Debian the flag -q than can be enabled in nc to get the data from Unix Sockets. Please see the comments in file socket-command.sh

   Docker recommends using CURL version 7.40 or higher to communicate with Unix Socket. Please update CURL version. Please see the comments in file socket-command.sh

3. TCP Socket: To troubleshoot TCP, make sure that the following commands returns a valid JSON output

   wget http://127.0.0.1:2375/containers/json -O - -q
   wget http://127.0.0.1:2375/containers/$CONTAINER_ID/stats -O - -q
   

   If you are executing the command from different machine other than the Docker Host Machine, please update the host part of the above url 127.0.0.1 to the IP of the Docker Host Machine. Also please update the port value 2375 if needed. Please refer to Prerequisites#3 section for more details. Please update $CONTAINER_ID accordingly
4. Metric Limit: Please start the machine agent with the argument -Dappdynamics.agent.maxMetrics=1000, if there is a metric limit reached error in the logs
5. Dashboard Upload: The dashboard upload works only with the SSL in Saas. If the machine agent connects to Saas servers on port 80, then you need to set these properties in the customDashboard section of config.yml. The properties are controllerPort: 443 and controllerSslEnabled: true. This is only application to AppDynamics Saas Environment.
6. Dashboard Upload: If there are any issues in uploading the dashboard, update the value of uploadDashboard to false in config.yml. This will create an xml in the logs directory. Please upload it manually through the controller ui. To debug any issues, attach it to the support ticket.
7. Recreate Dashboard: If the existing dashboard is deleted from the controller ui, then the extension will create a new dashboard.
8. Collect Debug Logs: Edit the file, <MachineAgent>/conf/logging/log4j.xml and update the level of the appender "com.appdynamics"  to debug

Workbench

Workbench is a feature that lets you preview the metrics before registering it with the controller. This is useful if you want to fine tune the configurations. Workbench is embedded into the extension jar.

1. Follow all the installation steps
2. Start the workbench with the command

   java -jar /path/to/MachineAgent/monitors/DockerMonitor/docker-monitoring-extension.jar

   This starts an http server at http://host:9090/. This can be accessed from the browser.
3. If the server is not accessible from outside/browser, you can use the following end points to see the list of registered metrics and errors.

   #Get the stats
   curl http://localhost:9090/api/stats
   #Get the registered metrics
   curl http://localhost:9090/api/metric-paths
   

4. You can make the changes to config.yml and validate it from the browser or the API
5. Once the configuration is complete, you can kill the workbench and start the Machine Agent

Metrics

The metrics will be reported under the tree Application Infrastructure Performance|$TIER|Custom Metrics|Docker

Please refer to this screenshot to view the complete list of metrics reported by the Docker Extension.

Known Issues

• The extension doesn't work with the newer versions Docker Unix Socket. Please enable the TCP socket. For details, please refer to the Troubleshooting #1

Support

Please contact help@appdynamics.com with the following details

1. config.yml
2. debug logs

Compatibility

Releases Notes

• 1.0.20: Server|Component:<TIER_NAME> is no longer supported by machine agent. Have to use Server|Component:<TIER_ID>
• 1.0.24: Defaulted the Null values to zero
• 1.0.25: External Library Update
• 1.0.26: Added CURL command to socket-command.sh