ZABBIX 2.4 – 6 Log file monitoring

Zabbix can be used for centralized monitoring and analysis of log files with/without log rotation support.

Notifications can be used to warn users when a log file contains certain strings or string patterns.

To monitor a log file you must have:

• Zabbix agent running on the host
• log monitoring item set up

Make sure that in the agent configuration file:

• 'Hostname' parameter matches the host name in the frontend
• Servers in the 'ServerActive' parameter are specified for the processing of active checks

Configure a log monitoring item:

Specifically for log monitoring items you must enter:

• The server and agent keep a trace of the monitored log's size and last modification time (for logrt) in two counters.
• The agent starts reading the log file from the point it stopped the previous time.
• The number of bytes already analyzed (the size counter) and last modification time (the time counter) are stored in the Zabbix database and are sent to the agent, to make sure it starts reading the log file from this point.
• Whenever the log file becomes smaller than the log size counter known by the agent, the counter is reset to zero and the agent starts reading the log file from the beginning taking the time counter into account.
• All files matching the filename format in the provided directory are analyzed every cycle the agent tries to get the next line from the log (for logrt).
• If there are several matching files with the same last modification time in the directory, then the agent will read lexicographically the smallest one.
• Zabbix agent processes new records of a log file once per Update interval seconds.
• Zabbix agent does not send more than maxlines of a log file per second. The limit prevents overloading of network and CPU resources and overrides the default value provided by MaxLinesPerSecond parameter in the agent configuration file.
• To find the required string Zabbix will process 4 times more new lines than set in MaxLinesPerSecond. Thus, for example, if a log[] or logrt[] item has Update interval of 1 second, by default the agent will analyse no more than 400 log file records and will send no more than 100 matching records to Zabbix server in one check. By increasing MaxLinesPerSecond in the agent configuration file or setting maxlines parameter in the item key, the limit can be increased up to 4000 analysed log file records and 1000 matching records sent to Zabbix server in one check. If the Update interval is set to 2 seconds the limits for one check would be set 2 times higher than with Update interval of 1 second.
• Additionally, log values are always limited to 50% of the agent send buffer size, even if there are no non-log values in it. So for the maxlines values to be sent in one connection (and not in several connections), the agent BufferSize parameter must be at least maxlines x 2.
• In the absence of log items all agent buffer size is used for non-log values. When log values come in they replace the older non-log values as needed, up to the designated 50%.
• For log file records longer than 256kB, only the first 256kB are matched against the regular expression and the rest of the record is ignored. However, if Zabbix agent is stopped while it is dealing with a long record the agent internal state is lost and the long record may be analysed again and differently after the agent is started again.
• Special note for “\” path separators: if file_format is “file\.log”, then there should not be a “file” directory, since it is not possible to unambiguously define whether ”.” is escaped or is the first symbol of the file name.
• Regular expressions for logrt are supported in filename only, directory regular expression matching is not supported.

Sometimes we may want to extract only the interesting value from a target file instead of returning the whole line when a regular expression match is found.

Since Zabbix 2.2.0, log items have the ability to extract desired values from matched lines. This is accomplished by the additional output parameter in log and logrt items.

output allows to indicate the subgroup of the match that we may be interested in.

So, for example

log[/path/to/the/file,large result buffer allocation.*Entries: ([0-9]+),,,,\1]

should allow returning the entry count as found in the content of:

Fr Feb 07 2014 11:07:36.6690 */ Thread Id 1400 (GLEWF) large result
buffer allocation - /Length: 437136/Entries: 5948/Client Ver: >=10/RPC
ID: 41726453/User: AUser/Form: CFG:ServiceLevelAgreement

The reason why Zabbix will return only the number is because output here is defined by \1 referring to the first and only subgroup of interest: ([0-9]+)

And, with the ability to extract and return a number, the value can be used to define triggers.