Centralized logging benefits more than just application troubleshooting. Today, many organizations are required to comply to regulatory standards like HIPAA, SOX, or FINRA, which means that application log control becomes even more important.

Few application infrastructures are built with good log handling functionality. Typically, applications just write log files to some location, possible roll log files according to date or size, and generally accumulate logging data on the application servers themselves. This quickly becomes unmanageable.

A centralized application logging solution drastically improves the capability to effectively control, mine, and exploit logging data. Here’s some advantages to using centralized logging over local log storage.

  1. Centralized logging helps keep your application disk partitions static and minimize disk I/O on your application servers. This usually save costs because you can limit the disk space your application servers need.
  2. By keeping storage requirements static, you also minimize you support costs. Having your support staff manually archive files is inefficient, error-prone and a waste of resources. Running out of disk space on your application servers is not a problem you should have to worry about.
  3. Centralized logs allow your support staff to do log research outside the production environment. Searching through log files using tools like “grep” should not steal disk IOPs from your application servers. It’s better to do the searching in a centralized location.
  4. Without centralized logging, it becomes a logistical nightmare to research a single transaction that may have been processed on any one of an array of app servers, since your support staff will have to log into each server and start searching through each.

Here’s some basic tips for centralized logging:

  1. It’s generally a good idea to keep your log files in the same timezone. GMT usually works best since you don’t have to worry about daylight savings time changes. Also you might have support staff in different timezones. Calculating a local offset from GMT is usually easier than from other timezones.
  2. Keep all your system clocks in sync using an NTP server or stratum 0 time keeping device such as a GPS device.
  3. You can often get away with less expensive storage for your log files. It depends on how much log crunching/parsing you do and how much disk I/O you need for that. Many times most of the intensive searching can be done non-interactively, so speed may be less important.
  4. For delivery you can use either the syslog protocol, or log delivery scripts. Unfortunately, syslog is not always reliable, but does offer the advantage of less disk I/O. There is some work (RFCs 3159, 5424) to make syslog more reliable as a protocol, but implementation is currently sparse. Log delivery scripts can work very well, though have a disadvantage of some local disk I/O to write the file to a local temporary storage spot and then read it back to write it to the central archive. A frequent delivery interval is good to limit the local temp space requirements; also it will increase the likelihood of the log file being cached when it’s written to the archive, which will minimize the disk I/O.
  5. For analysis, there are many off-the-shelf tools out there that can be implemented only with centralized logs. Any tool you write yourself will also benefit from having all the logs in one location.

© Copyright 2020 Rex Consulting, Inc. – All rights reserved

GMT Infrastructure Logs Script Server Timezone