AppDynamics Monitoring Extension for use with IBM Websphere MQ

Use case

Websphere MQ, formerly known as MQ (message queue) series, is an IBM standard for program-to-program messaging across multiple platforms.

The WebSphere MQ monitoring extension can monitor multiple queues managers and their resources ( queues ,channels and listeners). The metrics are extracted out using the PCF command messages.

The metrics reported for queue manager, queues,listeners and channel can be configured. The MQ Monitor currently supports IBM Websphere MQ version 7.x, 8.x and 9.x.

Prerequisites

1. This extension requires a AppDynamics Java Machine Agent installed and running.
2. If this extension is configured for Client transport type (more on that later), please make sure the MQ's host and port is accessible.
3. If this extension is configured for CLIENT transport type (more on that later), credentials of user with correct access rights would be needed in config.yaml (more on that later). If the hosting OS for IBM MQ is Windows, Windows user credentials will be needed.
4. This extension has been tested on Windows and Unix systems. Customers have successfully gotten it to work on AIX systems as well.

Dependencies

This extension has a dependency on the following seven JAR files from the IBM MQ distribution

com.ibm.mq.commonservices.jar
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
dhbcore.jar
com.ibm.mq.headers.jar
connector.jar
com.ibm.mq.pcf.jar

In newer versions of the MQ, IBM has removed connector.jar & dhbcore.jar and merged its contents in com.ibm.mq.allclient.jar.
These jar files are typically found in `/opt/mqm/java/lib` on a UNIX server but may be found in an alternate location depending upon your environment.
In case Client transport type, IBM MQ Client must be installed to get the MQ jars. To download IBM MQ Client jars, see here

Installation

1. Unzip contents of WMQMonitor-.zip file and copy to /monitors directory.
2. There are two transport modes in which this extension can be run
   1. Bindings : Requires WMQ Extension to be deployed in machine agent on the same machine where WMQ server is installed.
   2. Client : In this mode, the WMQ extension is installed on a different host than the IBM MQ server. Please install the IBM MQ Client for this mode to get the necessary jars as mentioned previously.
   Copy the following jars to the WMQMonitor directory

              com.ibm.mq.commonservices.jar
              com.ibm.mq.jar
              com.ibm.mq.jmqi.jar
              dhbcore.jar
              com.ibm.mq.headers.jar
              connector.jar
              com.ibm.mq.pcf.jar
       

   As mentioned previously, If you don't find the connector.jar & dhbcore.jar, please copy the com.ibm.mq.allclient.jar. If you are copying the com.ibm.mq.allclient.jar, please edit the monitor.xml to replace the connector.jar & dhbcore.jar entries with com.ibm.mq.allclient.jar
   If you don't want to copy the jar files, you can point to the jar files by providing the absolute paths to the jar files in the monitor.xml.
3. If you plan to use **Client** transport type,create a channel of type server connection in each of the queue manager you wish to query.
4. Edit the config.yaml file. An example config.yaml file follows these installation instructions.
5. Restart the Machine Agent.

Sample config.yaml: The following is a sample config.yaml file that depicts two different queue managers defined. The different fields are explained in the in-line comments.

#For most purposes, no need to change this.
    numberOfThreads: 10



    #This will create it in specific Tier aka Component. Replace . Please make sure to have a trailing |.
#To find out the COMPONENT_ID, please see the screen shot here  https://docs.appdynamics.com/display/PRO42/Build+a+Monitoring+Extension+Using+Java
    metricPrefix: "Server|Component:|Custom Metrics|WebsphereMQ|"
   
    encryptionKey: "SampleEncryptionKey"
    queueManagers:
      - host: "192.168.57.104"
        port: 1414
        #Actual name of the queue manager
        name: "TEST_QM_1"
        #Channel name of the queue manager
        channelName: "SYSTEM.ADMIN.SVRCONN"
        #The transport type for the queue manager connection, the default is "Bindings" for a binding type connection
        #For bindings type connection WMQ extension (i.e machine agent) need to be on the same machine on which WebbsphereMQ server is running
        #for client type connection change it to "Client".
        transportType: "Client"
        #user with admin level access, no need to provide credentials in case of bindings transport type, it is only applicable for client type
        username: "hello"
        password: "hello"

        #encryptedPassword: ""

        #This is the timeout on queue metrics threads.Default value is 20 seconds. 
        queueMetricsCollectionTimeoutInSeconds: 20

        queueFilters:
            #An asterisk on its own matches all possible names.
            include: ["*"]
            #exclude all queues that starts with SYSTEM or AMQ.
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM","AMQ"]

        channelFilters:
            #An asterisk on its own matches all possible names.
            include: ["*"]
            #exclude all queues that starts with SYSTEM.
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM"]

        listenerFilters:
            include: ["*"]
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM"]

      - host: "102.138.37.105"
        port: 1414
        #Actual name of the queue manager
        name: "TEST_QM_2"
        #Channel name of the queue manager
        channelName: "SYSTEM.ADMIN.SVRCONN"
        #The transport type for the queue manager connection, the default is "Bindings" for a binding type connection
        #For bindings type connection WMQ extension (i.e machine agent) need to be on the same machine on which WebbsphereMQ server is running
        #for client type connection change it to "Client".
        transportType: "Client"
        #user with admin level access, no need to provide credentials in case of bindings transport type, it is only applicable for client type
        username: "hello"
        password: "hello"

         #This is the timeout on queue metrics threads.Default value is 20 seconds. 
        queueMetricsCollectionTimeoutInSeconds: 20

        queueFilters:
            #Matches all queues  that starts with TACA..
            include: ["TACA*"]
            #exclude all queues that starts with SYSTEM or AMQ.
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM","AMQ"]

        channelFilters:
            #An asterisk on its own matches all possible names.
            include: ["*"]
            #exclude all queues that starts with SYSTEM.
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM"]

        listenerFilters:
            include: ["*"]
            exclude:
               - type: "STARTSWITH"
                 values: ["SYSTEM"]

    mqMertics:
      # This Object will extract queue manager metrics
      - metricsType: "queueMgrMetrics"
        metrics:
          include:
            - Status: "Status"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACF_Q_MGR_STATUS"

       # This Object will extract queue metrics
  - metricsType: "queueMetrics"
    metrics:
      include:
        - MaxQueueDepth: "Max Queue Depth"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_MAX_Q_DEPTH"
          ibmCommand: "MQCMD_INQUIRE_Q"
          
        - CurrentQueueDepth: "Current Queue Depth"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_CURRENT_Q_DEPTH"
          ibmCommand: "MQCMD_INQUIRE_Q"
          
        - OpenInputCount: "Open Input Count"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_OPEN_INPUT_COUNT"
          ibmCommand: "MQCMD_INQUIRE_Q"
          
        - OpenOutputCount: "Open Output Count"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_OPEN_OUTPUT_COUNT"
          ibmCommand: "MQCMD_INQUIRE_Q"

        - OldestMsgAge: "OldestMsgAge"
          ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACF_OLDEST_MSG_AGE"
          ibmCommand: "MQCMD_INQUIRE_Q_STATUS"

        - OnQTime: "OnQTime"
          ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACF_Q_TIME_INDICATOR"
          ibmCommand: "MQCMD_INQUIRE_Q_STATUS"

        - UncommittedMsgs: "UncommittedMsgs"
          ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACF_UNCOMMITTED_MSGS"
          ibmCommand: "MQCMD_INQUIRE_Q_STATUS"

        - HighQDepth: "HighQDepth"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_HIGH_Q_DEPTH"
          ibmCommand: "MQCMD_RESET_Q_STATS"

        - MsgDeqCount: "MsgDeqCount"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_MSG_DEQ_COUNT"
          ibmCommand: "MQCMD_RESET_Q_STATS"

        - MsgEnqCount: "MsgEnqCount"
          ibmConstant: "com.ibm.mq.constants.CMQC.MQIA_MSG_ENQ_COUNT"
          ibmCommand: "MQCMD_RESET_Q_STATS"

      # This Object will extract queue metrics
      - metricsType: "channelMetrics"
        metrics:
          include:
            - Messages: "Messages"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_MSGS"

            - Status: "Status"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_CHANNEL_STATUS" #http://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q090880_.htm

            - ByteSent: "Byte Sent"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_BYTES_SENT"

            - ByteReceived: "Byte Received"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_BYTES_RECEIVED"

            - BuffersSent: "Buffers Sent"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_BUFFERS_SENT"

            - BuffersReceived: "Buffers Received"
              ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_BUFFERS_RECEIVED"

    - metricsType: "listenerMetrics"
      metrics:
        include:
          - SessionCount : "SessionCount"
            ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_SESSION_COUNT"

          - Status : "Status"
            ibmConstant: "com.ibm.mq.constants.CMQCFC.MQIACH_LISTENER_STATUS"

Extension Working - Internals

This extension extracts metrics through PCF framework. A complete list of PCF commands are listed here.
Each queue manager has an administration queue with a standard queue name and the extension sends PCF command messages to that queue. On Windows and Unix platforms, the PCF commands are sent is always sent to the SYSTEM.ADMIN.COMMAND.QUEUE queue. More details about that is mentioned here.
By default, the PCF responses are sent to the SYSTEM.DEFAULT.MODEL.QUEUE. Using this queue causes a temporary dynamic queue to be created. You can override the default here by using the `modelQueueName` and `replyQueuePrefix` fields in the config.yaml. More details mentioned here.

Access Permissions

If you are in Bindings mode, please make sure to start the MA process under a user which has permissions to inquire,get,put (since PCF responses cause dynamic queues to be created) on the broker. Also, chg permission is also needed if the metrics (HighQDepth,MsgDeqCount,MsgEnqCount) related to reset_q_stats are enabled in the config.yaml . Similarly, for Client mode provide the credentials which have enough access permissions.

SSL Support

1. Configure the IBM SSL Cipher Suite in the config.yaml.
2. Note that, to use some CipherSuites the unrestricted policy needs to be configured in JRE. Please visit this link for more details.
3. To configure SSL, the MA's trust store and keystore needs to be setup with the JKS filepath. Please add the following JVM arguments to the MA start up command or script. -Dcom.ibm.mq.cfg.useIBMCipherMappings=false (If you are using IBM Cipher Suites, set the flag to true. Please visit this link for more details. -Djavax.net.ssl.trustStore=<PATH_TO_JKS_FILE> -Djavax.net.ssl.trustStorePassword=<PASS> -Djavax.net.ssl.keyStore=<PATH_TO_JKS_FILE> -Djavax.net.ssl.keyStorePassword=<PASS>

Metrics

The metrics will be reported under the tree Application Infrastructure Performance|$TIER|Custom Metrics|WebsphereMQ

Troubleshooting

Please check the /logs/machine-agent.log* for troubleshooting

1. Verify Machine Agent Data: Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics.
2. config.yaml:Validate the file here.
3. MQ Version incompatibilities: In case of any jar incompatibility issue, the rule of thumb is to Use the jars from MQ version 7.5. We have seen some jar incompatibility issues on IBM version 7.0.x ,version 7.1.x and version 8.x when the extension is configured in Client mode. However, after replacing the jars with MQ version 7.5's jars, everything worked fine.
4. The config cannot be null :
   This usually happens when on a windows machine in monitor.xml you give config.yaml file path with linux file path separator `/`. Use Windows file path separator `\` e.g. `monitors\MQMonitor\config.yaml` .
5. Metric Limit: Please start the machine agent with the argument -Dappdynamics.agent.maxMetrics=5000 if there is a metric limit reached error in the logs. If you don't see the expected metrics, this could be the cause.
6. Debug Logs:Edit the file, /conf/logging/log4j.xml and update the level of the appender com.appdynamics to debug . Let it run for 5-10 minutes and attach the logs to a support ticket

 

 

Password Encryption Support

To avoid setting the clear text password in the config.yml, please follow the process to encrypt the password and set the encrypted password and the encryptionKey in the config.yml

1. To encrypt password from the commandline go to/monitors/WMQMonitor dir and run the below common

   java -cp "websphere-mq-monitoring-extension.jar" com.appdynamics.extensions.crypto.Encryptor myKey myPassword

Workbench

Workbench is a feature by which you can preview the metrics before registering it with the controller. This is useful if you want to fine tune the configurations. Workbench is embedded into the extension jar.
To use the workbench

1. Follow all the installation steps
2. Start the workbench with the command

         java -jar /monitors/WMQMonitor/websphere-mq-monitoring-extension.jar
      

   
   This starts an http server at http://host:9090/. This can be accessed from the browser.
3. If the server is not accessible from outside/browser, you can use the following end points to see the list of registered metrics and errors.

   #Get the stats
       curl http://localhost:9090/api/stats
       #Get the registered metrics
       curl http://localhost:9090/api/metric-paths
   

4. You can make the changes to config.yml and validate it from the browser or the API
5. Once the configuration is complete, you can kill the workbench and start the Machine Agent.

Advance Troubleshooting

1. Error `Completion Code '2', Reason '2495'` :
   Normally this error occurs if the environment variables are not set up correctly for this extension to work MQ in Bindings Mode. If you are seeing `Failed to load the WebSphere MQ native JNI library: 'mqjbnd'`, please add the following jvm argument when starting the MA. -Djava.library.path=<path to lib64> For eg. on Unix it could -Djava.library.path=/opt/mqm/java/lib64
   Sometimes you may have to run the setmqenv script before using the above jvm argument to start the machine agent. . /opt/mqm/bin/setmqenv -s
   For more details, please check this doc. If you still see problems in Bindings mode, you may want to connect to WMQ server from a remote machine using Client mode. Make sure to provide Windows admin username and password in the config.yaml.
2. Error `Completion Code '2', Reason '2035'`:
   This could happen for various reasons but for most of the cases, for Client mode the user specified in config.yaml is not authorized to access the queue manager. Also sometimes even if userid and password are correct, channel auth (CHLAUTH) for that queue manager blocks traffics from other ips, you need to contact admin to provide you access to the queue manager.
   For Bindings mode, please make sure that the MA is owned by the mqm user. Please check this doc
3. Error MQJE001: Completion Code '2', Reason '2195'
   This could happen in Client mode. One way this could be fixed is to use 7.5.0.4 version of the IBM MQ jars.
4. The config cannot be null :
   This usually happens when on a windows machine in monitor.xml you give config.yaml file path with linux file path separator `/`. Use Windows file path separator `\` e.g. `monitors\MQMonitor\config.yaml` .

Support

Please contact help@appdynamics.com with the following details

1. config.yml
2. debug logs

Compatibility

Codebase

You can contribute your development ideas here.