AppDynamics Monitoring Extension for use with Kafka

AppDynamics Monitoring Extension for use with Kafka

An AppDynamics extension to be used with a stand alone Java machine agent to provide metrics for Apache Kafka

Use Case

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.


This extension extracts the metrics from Kafka using the JMX protocol. Make sure you have configured JMX in Kafka

To know more about JMX, please follow the below link

Troubleshooting steps

Before configuring the extension, please make sure to run the below steps to check if the set up is correct.

  1. Telnet into your Kafka server from the box where the extension is deployed.

          telnet <hostname> <port>
          <port> - It is the jmxremote.port specified.
          <hostname> - IP address

    If telnet works, it confirm the access to the Kafka server.

  2. Start jconsole. Jconsole comes as a utility with installed jdk. After giving the correct host and port , check if Kafka mbean shows up.

  3. It is a good idea to match the mbean configuration in the config.yml against the jconsole. JMX is case sensitive so make sure the config matches exact.

Metrics Provided

In addition to the metrics exposed by Kafka, we also add a metric called "Metrics Collected" with a value 0 when an error occurs and 1 when the metrics collection is successful.

Note : By default, a Machine agent or a AppServer agent can send a fixed number of metrics to the controller. To change this limit, please follow the instructions mentioned here. For eg.

    java -Dappdynamics.agent.maxMetrics=2500 -jar machineagent.jar


  1. Run "mvn clean install" and find the file in the "target" folder. You can also download the from AppDynamics Exchange.
  2. Unzip as "KafkaMonitor" and copy the "KafkaMonitor" directory to <MACHINE_AGENT_HOME>/monitors


Note : Please make sure to not use tab (\t) while editing yaml files. You may want to validate the yaml file using a yaml validator

  1. Configure the Kafka instances by editing the config.yml file in <MACHINE_AGENT_HOME>/monitors/KafkaMonitor/.
  2. Below is the default config.yml which has metrics configured already For eg.

    #This will create this metric in all the tiers, under this path
    metricPrefix: Custom Metrics|Kafka
    #This will create it in specific Tier/Component. Make sure to replace <COMPONENT_ID> with the appropriate one from your environment.
    #To find the <COMPONENT_ID> in your environment, please follow the screenshot
    #metricPrefix: Server|Component:<COMPONENT_ID>|Custom Metrics|Kafka
    # List of Kafka Instances
    - host: "localhost"
      port: 9999
      displayName: "Local Kafka Server"  #displayName is a REQUIRED field for level metrics.
    # number of concurrent tasks.
    # This doesn't need to be changed unless many instances are configured
    numberOfThreads: 10
    # The configuration of different metrics from various mbeans of Kafka server
    # For most cases, the mbean configuration does not need to be changed.
    #All MBeans which have attributes Count and MeanRate
    - mbeanFullPath: ["kafka.server:type=BrokerTopicMetrics,*",
            - Count: "Count"
            - MeanRate: "MeanRate"
    #All MBeans which have attributes Value
    - mbeanFullPath: ["kafka.server:type=DelayedOperationPurgatory,*",
            - Value: "Value"
  3. Configure the path to the config.yml file by editing the in the monitor.xml file in the <MACHINE_AGENT_HOME>/monitors/KafkaMonitor/ directory. Below is the sample For Windows, make sure you enter the right path.

         <!-- config file-->
         <argument name="config-file" is-required="true" default-value="monitors/KafkaMonitor/config.yml" />

Password Encryption Support

To avoid setting the clear text password in the config.yaml, please follow the process below to encrypt the password

  1. Download the util jar to encrypt the password from and navigate to the downloaded directory
  2. Encrypt password from the commandline java -cp appd-exts-commons-1.1.2.jar com.appdynamics.extensions.crypto.Encryptor encryptionKey myPassword
  3. Specify the passwordEncrypted and encryptionKey in config.yaml

Enable JMX

To enable JMX Monitoring for Kafka broker, please follow below instructions:

Edit and modify KAFKA_JMX_OPTS variable like below (please replace <> with your Kafka Broker hostname)

KAFKA_JMX_OPTS=" -Djava.rmi.server.hostname=< >"

Add below line in

 export JMX_PORT=${JMX_PORT:-9999}

Custom Dashboard


Always feel free to fork and contribute any changes directly via GitHub.


Kafka Versions Tested On0.10.1.0
Machine Agent Compatibility4.0+
Last Update19 Jan 2018


Releases Notes

  • 1.0.1: Initial release
  • 1.0.2:
    1. Fix for null in metric path if type or name property not present for a mbean.
    2. Added Workbench mode