AppDynamics Monitoring Extension for use with Solr

AppDynamics Monitoring Extension for use with Solr

 

This extension works only with the standalone machine agent.

 

Use Case

 

Solr is a popular open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search.
This extension collects metrics for all cores from Solr search engine and uploads them to AppDynamics Metric Broswer.

Solr statistics (Core, Query, Cache) are obtained through an HTTP request to the SolrInfoMBeanHandler http://<host>:<port>/solr/admin/mbeans and Memory statistics to SystemInfoHandler http://<host>:<port>/solr/admin/system. Please refer here for details.

 

Installation

 

  1. Download the file SolrMonitor.zip into <machineagent install dir>/monitors/
  2. Unzip the downloaded file
  3. Configure the extension by referring to the below section.
  4. Restart the Machine Agent.

In the AppDynamics Metric Browser, look for: Application Infrastructure Performance | <Tier> | Custom Metrics | Solr for default metric-path

 

Configuration

Note : Please make sure to not use tab (\t) while editing yaml files. You may want to validate the yaml file using a yaml validator

  1. Configure the Solr instance, Cores and Request handlers to monitor by editing the config.yml file in <MACHINE_AGENT_HOME>/monitors/SolrMonitor/. Below is the sample.

        #prefix used to show up metrics in AppDynamics
    metricPrefix: "Custom Metrics|Solr"
    
    #This will create it in specific Tier. Replace 
    #metricPrefix: "Server|Component:|Custom Metrics|Solr Monitor|"
    
    servers:
    - host: "localhost"
     port: 8983
     name: "Solr Monitor 1" #optional if only one server
     # Optional Parameters. Configure if any required
     username: ""
     password: ""
     contextRoot: ""
     usessl: ""
     proxyHost: ""
     proxyPort: ""
     proxyUsername: ""
     proxyPassword: ""
    - host: "localhost"
     port: 8983
     name: "Solr Monitor 2" #optional if only one server
     # Optional Parameters. Configure if any required
    #     username: ""
    #     password: ""
    #     contextRoot: ""
    #     usessl: ""
    #    proxyHost: ""
    #     proxyPort: ""
    #     proxyUsername: ""
    #     proxyPassword: ""
    
    # Example
    # cores
    # - name: "collection1"
    #     pingHandler: "/admin/ping"
    #     queryHandlers: ["/select", "/update"]
    #   - name: "collection2"
    #     pingHandler: "/admin/ping"
    #     queryHandlers: ["/admin/ping"]
    cores:
    - name: "gettingstarted"
     pingHandler: "/admin/ping"
     queryHandlers: ["/select", "/update"]
    
    numberOfThreads: 5
    

    Specify as many cores as you want to monitor and corresponding comma separated request handlers. If none of the cores are specified, default core with empty request handlers is monitored.

  2. Configure the path to the config.yml file by editing the in the monitor.xml file in the <MACHINE_AGENT_HOME>/monitors/SolrMonitor/ directory. Below is the sample

     <task-arguments>
         <!-- config file-->
         <argument name="config-file" is-required="true" default-value="monitors/SolrMonitor/config.yml" />
          ....
     </task-arguments>
    

 

Metrics

Note : By default, a Machine agent or a AppServer agent can send a fixed number of metrics to the controller. To change this limit, please follow the instructions mentioned here. For eg.

    java -Dappdynamics.agent.maxMetrics=2500 -jar machineagent.jar

By default, the metrics will be reported under the following metric tree:

Application Infrastructure Performance|Custom Metrics|$SERVERNAME|Solr Monitor

This will register metrics to all tiers within the application. We strongly recommend using the tier specific metric prefix so that metrics are reported only to a specified tier. Please change the metric prefix in your config.yaml

metricPrefix: "Server|Component:|Custom Metrics|Solr Monitor|"
For instructions on how to find the tier ID, please refer to the Metric Path subsection here - https://docs.appdynamics.com/display/PRO42/Build+a+Monitoring+Extension+Using+Java

Metrics will now be seen under the following metric tree:

Application Infrastructure Performance|$TIER|Custom Metrics|$SERVERNAME|Solr Monitor

 The following metrics are available for each core under Cores

Core Metrics

The following metrics are reported under <core>/CORE

Metric Name
Number of Docs
Max Docs
Deleted Docs

 

Query Statistics

The following metrics are reported under <core>/QUERYHANDLER for SearchHandler and UpdateHandler

Metric Name
Requests
Errors
Timeouts
Average Requests Per Minute
Average Requests Per Second
5 min Rate Requests Per Minute
Average Time Per Request (milliseconds)

 

Memory Statistics

The following metrics are reported under <core>/MEMORY/JVM

Metric Name
Used (MB)
Free (MB)
Total (MB)

The following metrics are reported under <core>/MEMORY/System

Metric Name
Free Physical Memory (MB)
Total Physical Memory (MB)
Committed Virtual Memory (MB)
Free Swap Size (MB)
Total Swap Size (MB)
Open File Descriptor Count
Max File Descriptor Count

 

Cache Statistics

The following metrics are reported under <core>/Cache /QueryResultCache

Metric Name
HitRatio %
HitRatioCumulative %
CacheSize (Bytes)

The following metrics are reported under <core>/Cache/DocumentCache

Metric Name
HitRatio %
HitRatioCumulative %
CacheSize (Bytes)

The following metrics are reported under <core>/Cache/FieldValueCache

Metric Name
HitRatio %
HitRatioCumulative %
CacheSize (Bytes)

The following metrics are reported under <core>/Cache/FilterCache

Metric Name
HitRatio %
HitRatioCumulative %
CacheSize (Bytes)

 

Troubleshooting

  1. Verify Machine Agent Data: Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics.
  2. config.yml: Validate the file here
  3. Metric Limit: Please start the machine agent with the argument -Dappdynamics.agent.maxMetrics=5000 if there is a metric limit reached error in the logs. If you don't see the expected metrics, this could be the cause.
  4. Check Logs: There could be some obvious errors in the machine agent logs. Please take a look.
  5. The config cannot be null error. This usually happenes when on a windows machine in monitor.xml you give config.yaml file path with linux file path separator /. Use Windows file path separator \ e.g. monitors\MQMonitor\config.yaml .
  6. Collect Debug Logs: Edit the file, /conf/logging/log4j.xml and update the level of the appender com.appdynamics to debug Let it run for 5-10 minutes and attach the logs to a support ticket
  7. You may see in the logs that all metrics have a value of 0. This usually happens when your Solr instance is corrupt or offline. Please restart your Solr instance in this situation and then restart the machine agent. The 'Ping Status' is a quick way to check whether or not the current Solr instance is up.

Custom Dashboard

 

 

 

Contributing

 

Always feel free to fork and contribute any changes directly via GitHub.

 

Support

 

For any support questions, please contact help@appdynamics.com.

 

Version:

1.1.2

Solr Version Tested:

6.5.0

Last Update:

05/12/2016

 

Release Notes:

1.0.5: Request Handlers configurable per each core

1.0.6 Moved configuration parameters from monitor.xml to config.yml and removed solr server pre-check

1.0.7 Fixed HTTP client library issue.

1.0.8 Added ping handler

1.0.9 Added support for multiple servers and general bug fixes

1.1.0 General bug fixes and improvements

1.1.2 Conversion to BigDecimal, conversion of memory metrics & additional unit tests