Monitoring Systems and Discipline
If you have a monitoring system, for maximum effectiveness, you need to be disciplined about handling alerts. “Handling alerts” means either A) fixing the problem that caused the alert in the case of a valid alert, or B) in case of false alarm, tuning the monitor system.
If you don’t handle the alerts promptly by either fixing the problem or fixing the alarm, three main things will happen:
- Problems won’t be handled promptly.
- Alarms/service checks won’t be adjusted and monitoring system’s accuracy and precision will skew.
- Many more alarms will go off and alerts will be sent, therefore reducing the significance of each one, causing people to start to ignore the ones that are redundant, and risking them then ignoring ones that are not.
If you can put resources and effort into keeping disciplined monitoring systems, your entire infrastructure will benefit.
One suggestion to achieve a monitoring system discipline would be to hire some experts like Rex Consulting to manage monitoring systems as a service. Another suggestion would be to have regular meetings to review false and redundant alarm notifications.