URL Monitoring Extension

An AppDynamics Machine Agent extension to visit a set of URLs and report whether they are up or down (and optionally whether certain text patterns appear on those pages).

This extension requires the Java Machine Agent.

Installation

1. Download and unzip UrlMonitor-<version>.zip into $MACHINE_AGENT_HOME/monitors. This will create a new directory called UrlMonitor.
2. Copy UrlMonitor directory where you installed the machine agent, under $AGENT_HOME/monitors.
3. In $AGENT_HOME/monitors/UrlMonitor, edit the configuration files (monitor.xml and config.yaml) to configure the plugin.
4. Restart the machine agent.

Configuration

Every AppDynamics extension has amonitor.xmlfile that configures the extension. In this case, themonitor.xmlfor this extension just has a single option: the path where the extension can find the mainconfig.yamlfile. Note that the path is relative to$AGENT_HOME.

   <task-arguments>
      <argument name="config-file" is-required="true" default-value="monitors/UrlMonitor/config.yaml" />
    </task-arguments>

 

The main configuration for this extension then lives in a file called config.yaml. It uses a simple syntax that anyone can edit with a simple text editor. Note: Please avoid using tab (\t) when editing yaml files. You may want to validate the yaml file using a yaml validator.  Here's a sample:

 
clientConfig:
    maxConnTotal:             1000
    maxConnPerRoute       1000
    maxRedirects:               10
    ignoreSslErrors:          true
    userAgent:                Mozilla/5.0 AppDynamics-UrlMonitor/1.0.6

defaultParams:
    method:                   GET
    socketTimeout:            30000
    connectTimeout:           30000
    numAttempts:              3
    treatAuthFailedAsError:   true

#Sites that need to be monitored
sites:

     #No authentication, with a pattern to match
   - name:     Google
     url:      http://www.google.com
     followRedirects: false
     groupName: MySites
     # Patterns to be matched, multiple patterns(to be matched) can be configured for a given site
     matchPatterns:
     - name: LuckyButton
       type: caseInsensitiveSubstring
       pattern: Google

   - name:     AppDynamics
     url:      http://www.appdynamics.com
     authType: BASIC

   - name:     File Download
     url:      https://github.com/Appdynamics/url-monitoring-extension/releases/download/1.0.6/UrlMonitor.zip

    # Basic Authentication with password encryption
   - name:       My Controller
     url:        https://mycontroller.saas.appdynamics.com/controller/rest/applications
     username:   demouser@customer1
     password:   welcome
     encryptedPassword: "IGVtC9eudmgG8RDjmRjGPQ=="
     encryptionKey: 
     authType: BASIC

     #NTLM Auth Sample Configuration
   - name:     My Controller
     url:      http://localhost:8090/controller
     username: user@DOMAIN
     password: password
     authType: NTLM
     connectTimeout: 60000

     # Client Cert Auth Sample Configuration
   - name:         LocalHost
     url:          https://localhost:8443
     password:     password
     authType:     SSL
     keyStoreType: SUNX509
     keyStorePath: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/client.jks
     keyStorePassword: password
     trustStorePath: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/client.jks
     trustStorePassword: password

     #POST request sample configuration
   - name:     My POST site
     url:      http://localhost:8293/api/v1/metrics
     username:
     password:
     connectTimeout: 60000
     method:   POST
     headers:
           Content-Type: application/json
     requestPayloadFile: src/test/resources/conf/postrequestPayloadFile.json
     matchPatterns:
       - name:       Error
         type:       substring
         pattern:    Error 400

     #Proxy Configuration
   - name:     Google
     url:      http://www.google.com
     groupName: MySites
     proxyConfig:
       host: ""
       port: ""
       username: ""
       password: ""

#prefix used to show up metrics in AppDynamics. This will create it in specific Tier. Replace
metricPrefix: Server|Component:|Custom Metrics|URLMonitor|
#This will create this metric in all the tiers, under this path
#metricPrefix: Custom Metrics|URLMonitor|

Examples

Increase the timeout threshold for a site that is often slow:

- name:             My Slow Site
  url:                 http://www.wordpress.com
  connectTimeout:   60000

Supply a username and password for HTTP Basic authentication:

- name:            My Login Page
  url:             http://localhost:8090/controller/rest/applications
  authtype:        BASIC
  username:        demouser@customer1
  password:        welcome

Retrieve the Google home page and make sure the "I'm Feeling Lucky" button is visible:

- name:           Google
  url:                http://www.google.com
  matchPatterns:
    - name:       LuckyButton
      type:       substring
      pattern:    I'm Feeling Lucky

Retrieve the Google home page and count how many times the word "Mail" appears:

- name:           Google
  url:            http://www.google.com
  matchPatterns:
    - name:       MailCount
      type:       word
      pattern:    Mail

POST xml or json payload to any url and search for the patterns in the response

 - name:     My POST site
    url:      http://myposturl
    method:   POST
    headers:
          Content-Type: application/json
    requestPayloadFile: path/to/postrequestPayloadFile.json
    matchPatterns:
        - name:       Error
          type:       substring
          pattern:    Error 400

Configuration Reference

Client Section

The clientConfig section sets options for the HTTP client library, including:

Option NameDefault ValueOption Description
maxConnTotal1000Maximum number of simultaneous HTTP connections
maxConnPerRoute1000Maximum number of simultaneous HTTP connections to a single host
threadCount10Maximum number of Threads spawned to cater HTTP request
ignoreSSlErrorsfalseWhether to ignore errors in SSL certificate validation or host validation
userAgentMozilla/5.0 (compatible; AppDynamics UrlMonitor;http://www.appdynamics.com/)Custom User-Agent header to send with requests (can be used to mimic desktop or mobile browsers)
followRedirectstrueWhether the client should follow Redirect responses
maxRedirects10Maximum redirects
Default Params Section

The defaultParams section sets the default options for all sites. These options can then be overriden at the individual site level.

Option NameDefault ValueOption Description
methodGETHTTP method to use (e.g. GET, POST, HEAD, OPTIONS, etc.).
socketTimeout30000Maximum time to wait for a socket connection to open, in milliseconds
connectTimeout30000Maximum time to wait for the HTTP handshake, in milliseconds
numAttempts1Number of times the site will be retrieved. The metrics then reported will be an average over all attempts.
treatAuthFailedAsErrortrueIf false, the extension will report the site status as "SUCCESS" even if authentication fails.
Site Section
Option NameDefault ValueMandatoryOption Description
namenoneyesName of the url with which metric folder that will be created in Metric Browser
urlnoneyesThe url to monitor
groupNamenonenoThe group under which site needs to be categorised
authTypenonenotype of authentication, supported auth are Basic, NTLM, Client Cert
matchPatternsnonenoMatches the specified patterns in the URL response , and reports the total number of matches count as metric
proxyConfignullnoSpecify the host and port of the proxy
headersnonenoComponent of request header section, e.g.: Content-Type
requestPayloadFilenonenoPayload file(XML or JSON) to upload to URL
ProxyConfig section
Option NameDefault ValueMandatoryOption Description
hostnoneYes(if proxy config specified)proxy host
portnoneYes(if proxy config specified)proxy port
usernamenoneYes(if proxy config specified)proxy username
passwordnoneYes(if proxy config specified)proxy password
Auth Type
Option NameDefault ValueMandatoryOption Description
authTypenoneYes(if authType is specified))Name of the authentication type: BASIC, NTLM, ClientCert
usernamenullYes(if authType is specified))username
passwordnullYes(if authType is specified))password
encryptedPasswordnonenoencrypted password if using password encryption
encryptionKeynonenothe key used to encrypt the password
keyStoreTypenonenokeyStoreType, used only in Client Cert Auth
keyStorePathnonenopath to keyStore file, used only in Client Cert Auth
keyStorePasswordnonenokeyStorePassword, used only in Client Cert Auth
trustStorePathnonenopath to trustStore file, used only in Client Cert Auth
trustStorePasswordnonenotrustStorePassword, used only in Client Cert Auth
Match Pattern Section
Option NameDefault ValueMandatoryOption Description
namenoneYes(if MatchPattern specified)Name of the metric folder that will be created in Metric Browser
patternnoneYes(if MatchPattern specified)The string to search for
typesubstringYes(if MatchPattern specified)Can be one of: substring, caseInsensitiveSubstring, regex, or word (see below)

 

The options for the pattern type are:

ValueMeaning
substringExact match on the given string
caseInsensitiveSubstringCase-insensitive match on the given string
regexRegular expression match
wordCase-insensitive, but the target string must be surrounded by non-word characters

Metrics for match pattern appears under the following path:
Site->Pattern Matches -> Name of MatchPattern(As specified in config.yml) -> Count

Password Encryption Support

To avoid setting the clear text password in the config.yml, please follow the process to encrypt the password and set the encrypted password and the key in the config.yml

  1. Download the util jar to encrypt the password from here
  2. Encrypt password from the commandline java -cp "appd-exts-commons-1.1.2.jar" com.appdynamics.extensions.crypto.Encryptor myKey myPassword
  3. These values should be used in the passwordEncrypted and encryptionKey fields in config.yml

Metrics Provided

In the AppDynamics Metric Browser, URL Monitor's metrics can be seen at: Application Infrastructure Performance | Tier-ID | Custom Metrics | URL Monitor   Following metrics are reported for each site:

  • Average Response time (ms) -> The time after the request is sent until the first byte is received back.
  • First Byte Time (ms) -> Time taken from the time the request build has started to receive the first response byte.
  • Download Time (ms) -> Total time taken to receive the entire response from the URL.
  • Response Bytes -> It represents the length of the response returned from the URL.
  • Response Code -> It represents the HTTP status code returned from the URL.
  • Status -> It represents whether the URL is FAILED(2), ERROR(3) or SUCCESS(4). Possible values are: UNKNOWN(0), CANCELLED(1), FAILED(2), ERROR(3), SUCCESS(4)
  • Responsive Count(Available at GroupName Level) -> Number of sites in a given group, that responded successfully.

Workbench

Workbench is a feature by which you can preview the metrics before registering it with the controller. This is useful if you want to fine tune the configurations. Workbench is embedded into the extension jar.
To use the workbench
  1. Follow all the installation steps
  2. Start the workbench with the command
          java -jar /monitors/URLQMonitor/url-monitoring-extension.jar
       
    This starts an http server at http://host:9090/. This can be accessed from the browser.
  3. If the server is not accessible from outside/browser, you can use the following end points to see the list of registered metrics and errors.
        #Get the stats
        curl http://localhost:9090/api/stats
        #Get the registered metrics
        curl http://localhost:9090/api/metric-paths
    
  4. You can make the changes to config.yml and validate it from the browser or the API
  5. Once the configuration is complete, you can kill the workbench and start the Machine Agent.

Custom Dashboard

url-pinger-dashboard.png

Contributing

Always feel free to fork and contribute any changes directly via GitHub

Community

Find out more in the Community

Support

For any support questions, please contact help@appdynamics.com.

 

Version:

1.2.7

Compatibility:

3.7 or later

Last Update:

11 October 2017

 

 1.0.2 - Fixed Memory leak

 1.0.3 - Fixed thread pool shutdown issue

 1.0.4 -  Using ning async library.adding more flexibility, features and metrics.

 1.1.0 - Added Regex Matching Patterns

 1.1.1 - Fixing the typo in the pom for config.yml file

 1.1.2 - Fixing bug in response status and adding proxy support

 1.2.0 - Making sure the Machine agent is provided and is not bundled in the jar

 1.2.1 - Support for POST and change in timerollup_type to average for some metrics

 1.2.2 - ignoreSSLErrors functionality

 1.2.3 - Fixed metric drop issue in case of large number of URLs

 1.2.4 - Adding groupName to group multiple sites

 1.2.5 - Added support for NTLM auth and ignoring SSL Cert errors

 1.2.6 - Added support for Client Side Cert auth and password encryption

 1.2.7 - Enhancement: followRedirect added at Site level;Fixed config.yml path related bug