Adding an additional layer of filtering through NXLOG
Have you ever wanted to apply an additional filtering layer to your logs before sending them to Nagios Log Server (NLS)? If you have are sending streams of logs to NLS but know that certain logs will not be used you can use NXLOG to mitigate data usage by filtering out the unnecessary logs. One method for doing this is as follows:
- Set up NXLOG if you have not already done so. The download and setup instructions can be found at https://nxlog.co/.
- Setup nxlog.conf. This file should be placed inside the nxlog/conf folder and may already exist. The way nxlog.conf works is by designating an input tag. Within those tags we need to specify a module to tell nxlog how you want the input source to be separated by fields. So if we wanted to filter out some logs from the windows event logs we can do the following:
<Input eventlog> Module im_msvistalog </Input>
Note that without specifying a module the NXLOG service will not be able to run or be restarted because NXLOG will not know how to parse the log fields.
- Add in your filter by specifying a regex. NXLOG gives the user scripting capabilities in the form of the NXLOG language (which is very similar to PERL syntactically). To filter using regex we add in an Exec command which essentially goes through the input line by line and executes a script. For example, if you wanted to filter out any event logs that mentioned ‘Read’ inside the raw_event field and only for EventID 400 you would do the following:
Exec if $EventID == 400 and ( $Message =~ /Read/m ) drop ();
This will execute the script on each log input. The input will be parsed line by line and if a regex match is found and the EventID matches it will drop the entire log.
- Send the output to NLS. This is achieved by specifying NLS within the <Output> </Output> tags.
Your input tags should look like this :
<Input eventlog> Module im_msvistalog Exec if $EventID == 400 and ( $Message =~ /Read/m ) drop (); </Input>
Now everything that matches the conditional within the Exec if statement will result in the logs being tossed and now more useful data can be forwarded to NLS instead.