Apache Log Monitoring Extension

Apache Log Monitoring Extension

 

Use Case

Monitors Apache access log file and reports metrics such as successful hits, bandwidth and page access count of visitors, spiders, browsers and operating systems.

 

Has the ability to display individual metrics per visitor, spider, browser, operating system, response code and page request.

 

This extension works only with the standalone machine agent.

 

Note: By default, the Machine agent can only send a fixed number of metrics to the controller. This extension can potentially report thousands of metrics, so to change this limit, please follow the instructions mentioned here.

 

Installation

  1. Download and unzip ApacheLogMonitor-<version>.zip into <machine_agent_dir>/monitors/
  2. Edit config.yaml file in ApacheLogMonitor/conf and provide the required configuration (see Configuration section)
  3. Restart the Machine Agent.

 

Configuration

 

config.yaml

Note: Please avoid using tab (\t) when editing yaml files. You may want to validate the yaml file using a yaml validator.

 

ParamDescriptionDefault ValueExample
nameThe alias name of this log, used in metric path. "Staging Apache"
logDirectoryThe full directory path of the access log "/var/log/apache2"
logNameThe access log filename. Supports wildcard (*) for dynamic filename Static name: 
"access.log"

Dynamic name: 
"access*.log"
logPatternThe grok pattern used for parsing the log. See examples for pre-defined pattern you can use.

If you're using a custom log format, you can create your own grok pattern to match this, see Grok Expressions section.
 "%{COMMONAPACHELOG}" - for common log format

"%{COMBINEDAPACHELOG}" - for combined log format
hitResponseCodesThe response codes used to determine a successful hit. Leave null to use default values.200, 304200, 201, 304, 305
nonPageExtensionsThe URL extensions used to determine if request is for non-page access, e.g. image. Leave null to use default values"ico", "css", "js",
"class","gif","jpg",
"jpeg","png","bmp",
"rss","xml","swf"
"pdf","jpg"
metricsFilterForCalculationFilters unwanted metrics----------
excludeVisitorsThe list of visitor hosts to exclude. Note, this supports regex, so period must be escaped. Specific Host:
"10\\.10\\.10\\.5",
"127\\.1\\.1\\.0"

Host Regex:
"10.*", "127.*"
excludeSpiders*The list of spider names to exclude. Note, this supports regex. Specific Spider:
"GoogleBot",
"Yahoo"

Spider Regex:
"Google.*"
excludeUrlsThe list of request URLs (any files) to exclude. Note, this supports regex. Specific URL:
"/test.html",
"/test/admin.html"

URL Regex:
"/test/.*"
excludeBrowsers*The list of browser names to exclude. Note, this supports regex. Specific URL:
"Chrome",
"Safari"

URL Regex:
"Chro.*"
excludeOs*The list of OS names to exclude. Note, this supports regex. Specific OS:
"MAC OS X"

OS Regex:
"MAC.*"
individualMetricsToDisplayDisplays individual metrics----------
includeVisitorsThe list of visitor hosts to display. Note, this supports regex, so period must be escaped. Specific Host:
"10\\.10\\.10\\.5",
"127\\.1\\.1\\.0"

Host Regex:
"10.*", "127.*"
includeSpiders*The list of spider names to display. Note, this supports regex. Specific Spider:
"GoogleBot",
"Yahoo"

Spider Regex:
"Google.*"
includePagesThe list of pages to display. Note, this supports regex. Specific Page:
"/test.html",
"/test/admin.html"

Page Regex:
"/test/.*"
includeBrowsers*The list of browser names to display. Note, this supports regex. Specific URL:
"Chrome",
"Safari"

URL Regex:
"Chro.*"
includeOs*The list of OS names to display. Note, this supports regex. Specific OS:
"MAC OS X"

OS Regex:
"MAC.*"
includeResponseCodes*The list of response codes to display. 200, 304, 404 500
--------------------
noOfThreadsThe no of threads used to process multiple apache logs concurrently33
metricPrefixThe path prefix for viewing metrics in the metric browser."Custom Metrics|Apache Log Monitor|""Custom Metrics|Apache Log Monitor2|"

 

*Requires user-agent details in the log, e.g. use combined log pattern in apache + specify logPattern as "%{COMBINEDAPACHELOG}" in this config.yaml.

 

sample config.yaml with static filename and dynamic filename

apacheLogs:
  - name: "StaticName"
    logDirectory: "/var/log/apache2"
    logName: "access.log"
    logPattern: "%{COMMONAPACHELOG}"
    hitResponseCodes: [ ] #leave null to use default values
    nonPageExtensions: [ ] #leave null to use default values

    metricsFilterForCalculation:
       excludeVisitors: [ ]
       excludeSpiders: [ ]
       excludeUrls: [ ]
       excludeBrowsers: [ ]
       excludeOs: [ ]

    individualMetricsToDisplay:
       includeVisitors: ["10\\.10.*" ]
       includeSpiders: ["Google.*" ]
       includePages: ["/test/.*" ]
       includeBrowsers: ["Chrome.*" ]
       includeOs: ["MAC.*" ]
       includeResponseCodes: [200, 305, 304, 400, 401, 500 ]

  - name: "DynamicLog"
    logDirectory: "/usr/log/apache2"
    logName: "access*.log"
    logPattern: "%{COMBINEDAPACHELOG}"
    hitResponseCodes: [ ] #leave null to use default values
    nonPageExtensions: [ ] #leave null to use default values

    metricsFilterForCalculation:
       excludeVisitors: [ ]
       excludeSpiders: [ ]
       excludeUrls: [ ]
       excludeBrowsers: [ ]
       excludeOs: [ ]

    individualMetricsToDisplay:
       includeVisitors: [ ]
       includeSpiders: [ ]
       includePages: [ ]
       includeBrowsers: [ ]
       includeOs: [ ]
       includeResponseCodes: [ ]

noOfThreads: 3        

metricPrefix: "Custom Metrics|Apache Log Monitor|"

 

Grok Expressions

Grok is a way to define and use complex, nested regular expressions in an easy to read and use format. Regexes defining discrete elements in a log file are mapped to grok-pattern names, which can also be used to create more complex patterns.

 

Grok file is located in ApacheLogMonitor/conf/patterns/grok-patterns.grok.

 

To add your own custom grok expression, simply edit the file above. Note, you must ensure that mandatory fields are captured:

  • clientip
  • response
  • bytes
  • request

Optional field used to determine the browser, os and spider details is as follow:

  • agent

For example:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA&colon;rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

MYCUSTOMAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

 

You can use Grok Debugger to validate your expression.

 

Then, define your custom expression in logPattern field in config.yaml, e.g.

...
logPattern: "%{MYCUSTOMAPACHELOG}"
...

 

Metrics

 

Definition

MetricDescription
HitsNo of any file requests where response code matches the defined hitResponseCode
Bandwidth (bytes)File size in bytes
PagesThe no of page requests, excluding files where extensions are defined in nonPageExtensions.

 

Typical Metric Path: Application Infrastructure Performance|<Tier>|Custom Metrics|Apache Log Monitor|<Log Name>| followed by the individual categories/metrics below:

 

MetricDescription
Total HitsOverall Total Hits (Visitor Hits + Spider Hits)
Total Bandwidth (bytes)Overall Total Bandwidth (Visitor Bandwidth + Spider Bandwidth)
Total PagesOverall Total Pages (Visitor Pages + Spider Pages)

 

Visitor, Spider, OS and Browser

MetricDescription
Total HitsNo of hits
Total Bandwidth (bytes)Bandwidth size
Total PagesNo of Pages

 

Page

MetricDescription
Total HitsNo of hits
Total Bandwidth (bytes)Bandwidth size

 

Response Code

MetricDescription
HitsNo of times this response code is returned for any file request
Bandwidth (bytes)Bandwidth size
PagesNo of times this response code is returned for any page request

 

Custom Dashboard Example

sample_custom_dashboard.png

 

Contributing

Always feel free to fork and contribute any changes directly via GitHub.

 

Support

For any questions or feature request, please contact AppDynamics Center of Excellence.

 

Version:1.0.1
Agent Compatiblity3.7+
Last Update25 June 2015