BlackDog Foundry Bookmark This page

In addition to the built-in log formats, LogDiver allows you to define your own custom log file formats using regular expressions.

This tutorial describes how to create your own custom log file parser.

Example Log File Format

For the same of the exercise, let’s assume that we have some application that produces log files called output-yyyy-mm-dd.log that look like the following:

timestamp severity returnCode customerName description

Each field is described below:

Field Description
timestamp The time that the event occurred
severity A code that indicates what severity level this line represents. Can be one of the following values:
  • DBG – Debug
  • ERR – Error
  • AUD – Audit
returnCode A simple numeric value indicating a return code
customerName The logged on user, with containing double quotes
description A description of the actual activity. May wrap over multiple lines

Some examples of log lines might be:

2012-12-22 12:34:56.100 DBG 0 "John Smith" Logged in
2012-12-22 12:34:56.200 DBG 0 "John Smith" Viewed page X
2012-12-22 12:34:56.300 ERR 2 "John Smith" Database not accessible:
Reason: File system full
Server: PRD-1
2012-12-22 12:34:56.400 AUD 1 "Peter Dark" Revoked access

Adding New Extractor

The first thing that you need to do is to add a new extractor using the Extractor Editor. To do this, click on the Import Log File toolbar item, and then click on the Extractor Editor icon (Extractor Editor icon).

Opening the Extractor Editor

This will open the Extractor Editor window, where you can add/edit/remove extractors. Add a new Extractor by clicking on the + button at the bottom of the the list.

Basic Settings

Give your new extractor a name, and modify the default date format to match the format using the Unicode Technical Standard #35.

Setting the name and date format

Setting the Pattern

Now, you need to describe the layout of each line. The pattern is expected to be a valid regular expression that uses the group syntax to identify fields.

To get you started, I have already identified the regular expression for the sample file as:

^(\S+ \S+) (\S+) (\S+) (\"[^\"]*\") (.*)

A very quick summary of this pattern is as follows:

  • Two words (date/time)
  • One word (severity)
  • One word (return code)
  • A double-quote followed by a bunch of chars (that aren’t double-quotes) up to the next double-quote
  • Everything else

A really useful utility that I use to create my regular expressions is RegExRX – see screenshot below:

RegExRX

As you tab out of the pattern field, you will notice that the Fields table gets a number of rows added – one for each of your regex groups. This table is editable, so you can choose which regex group gets assigned to which LogDiver field.

Fields

The Custom Description allows to you “rename” the column in the main event viewer to be more relevant to your specific business intent. In the example above, I have renamed authId to be User Name.

The Visible flag indicates whether this field should be displayed on the main event viewer by default. You can hide/show columns at any time, but this allows you to initially hide fields that may not be as important to you.

Mapping Severity

The next task is to map the input severity text to a specific LogDiver severity level (so you can easily and consistently filter by severity).

Severity map

Matching Files and Multiple Lines

Technically speaking, you are almost done at this point, but there are a couple of extra things you can do to make your extractor a little friendlier.

When you choose a file in the Import Log File dialog, LogDiver will automatically filter the list of available extractors based on the filename (you can always see the full list by unchecking the Only include suggested file handlers radio button). Usually, based on the file name, LogDiver is able to determine which is the best extractor to use.

In the example describe in this tutorial, the naming convention for the log files is output-yyyy-mm-dd.log, so you could be as crude as using a pattern like output-.*\.log, or you could be quite sophisticated and use a pattern like output-\d\d\d\d-\d\d-\d\d\.log.

And lastly, in order to handle log files that may have embedded newlines in them (like the above example), you can specify a pattern that LogDiver uses to match on to determine where a new event starts vs where the previous one continues.

Without specifying a New event regex pattern, the 4th and 5th lines would just be ignored because they doesn’t match the regular expression for a log event that you defined earlier. To remedy this, and to keep things simple, we’ll just say that if the line starts with 2012 then it is a new event, otherwise, the line will be treated as a continuation of the previous event.

Matching files and new events

Save and close the Extractor Editor.

Using Your New Extractor

Now when you select a file that matches your file name pattern, LogDiver will automatically select your extractor.

Auto-selecting extractor

And after clicking on Open, your main LogDiver window should look like:

Logdiver descriptor events

And that is that! Easy as pie.

Categories

Copyright © 2012 BlackDog Foundry