The Ultimate Manual For Alert Fatigue
Alert fatigue can cause significant errors for those caring for patients at a hospital or for those monitoring security issues in an organization’s network.
When people receive too many alerts and alarms that don’t indicate actual serious situations, they tend to begin ignoring or dismissing all of the alerts. This often leads to their missing a truly crucial alert, leading to a devastating error.
What is Alert Fatigue Anyway?
Alert fatigue is what happens when people receive so many alerts they become desensitized to them. This especially happens when the same alerts go out for all situations, from those that simply need monitoring to those that require immediate or emergency response.
If the alerts for less-important issues make up 99% of all of the alerts the person receives, alert fatigue often causes that person to miss or ignore when the 1-in-100 extremely serious alert occurs that requires immediate attention.
Most Common Victims: Hospitals And Security Networks
One of the areas where alert fatigue occurs most frequently is in hospitals and other medical facilities. When medical measuring equipment uses settings that contain a level of sensitivity that’s too high, false alarms become frequent.
If medical personnel experience alert fatigue due to this frequency of false alarms, they could miss a significant medical issue in the hospital that could lead to additional illness, worsening symptoms, or even death for a patient.
A similar thing can occur with network security issues in an organization. If the measurement settings that trigger alerts about network security issues are too sensitive, the system will generate a large number of false alarms. This makes alert fatigue a strong possibility for those tasked with monitoring the network.
Alert fatigue on the organization’s network could cause the security team to miss a significant security breach, assuming it’s another false alarm, leading to devastating results for the organization.
How Combatting Alert Fatigue Works
To reduce alert fatigue, organizations will want to reduce the number of false alarms the system generates. When team members know that the alert is a legitimate one the majority of the time, they will be more likely to respond to the alert, rather than ignoring it and hoping for the best.
Repeated false alarms are one of the most common contributors to alert fatigue, especially in security networks. A recent Cloud Security Alliance survey showed that almost one-third of security professionals ignore security alerts because of alert fatigue from too many false alarms.
Employees responding to alarms must stop their current tasks to check on the alarm. If false alarms happen several times during the workday, employees have difficulty focusing on other tasks because of the constant interruptions.
Eventually, these employees may begin slowing their responses to the alerts or even ignoring them entirely, assuming they’re all false alarms. This could cause the employees to miss a real alarm or assume it’s false when it isn’t.
Taking clear steps to reduce the number of false alarms will be a primary step in reducing alert fatigue.
Example 1: Create System Data Filters
Creating filters that sort the levels of alerts that the system generates is an important step in reducing alert fatigue. Some of the steps to follow include:
- Basic Filter: As the raw data regarding security alerts enters the system, apply a filter to the data. The filter should use the data that false alarms generated for the network in the past to discern between false and real alarms. This should eliminate a large number of false alarm events immediately.
- Behaviors: The system can use AI to learn the behaviors of members of the network, and determine which events are typical behavior and which actually require an alert.
- Profile History: The system can study the history of individual users on the network. Should the history show that an odd event actually marks a normal behavior for the user, rather than a malicious event, the system can choose to ignore it.
After applying the various filters, any events that remain will result in alerts for the team. These filters could drop the number of alerts sent to the team by 90% or more, which will reduce the chances of alert fatigue.
Example 2: Create Individual Alert Levels
If the security plan for the organization requires that every event must generate some sort of alert, this can lead to significant alert fatigue for members of the security team.
In a case like this, it will be beneficial to develop a rating system, a color-coding system, or different sounds to give the security team members an idea of the severity of the event. The system should be able to automate these severity levels, giving team members an idea of whether they need to immediately respond or whether the alert is more for informational purposes.
Alerts that would carry an information-only level allow team members to watch a situation to see if it alleviates or worsens. Human intervention for the alert only occurs if the situation worsens.
If the organization is going to rely on alert levels, it’s important to set them up properly, so the system doesn’t misidentify important alerts that need immediate human intervention.
Take the time to study the past history of alerts that the system generates. Base the alert system on that information for the best and safest results.
Example 3: Understand How Alerts Relate to Each Other
When the system generates an alert, it’s important for team members to have context about the particular alert and about anything else going on in the system.
It’s possible that one particular alert could be a piece of a puzzle that shows a greater likelihood of a significant attack. However, if the security team only has information about the single alert and has no information about the other occurrences in the system, the team may miss the impending attack.
Along those same lines, an individual alert that pops up without context may seem extremely serious. However, if the team members have all of the information, they may realize that the alert doesn’t require immediate attention.
One solution to this problem would involve sending all alerts and corresponding information to team members, letting them sort out which information is important to determine the severity of the event and the alert. However, this could lead to significant alert fatigue, defeating the purpose of giving the team members the extra information in the first place.
A better option would be to train the system to learn when a certain type of alert requires extra information to determine its true level of severity. The system may be able to automate the process of checking for the extra information on its own. If the system finds that the alert and the extra information warrant human intervention, it then can generate an alert for the team.
Another option could be to clearly determine who needs to receive alerts and who doesn’t, both in terms of department context and chain of command. A minor alert on a situation that needs monitoring doesn’t need to go to the entire company or even the entire department. An alert that the building is on fire, on the other hand, needs to go to everyone!
These processes would cut down on the number of alerts a team or company receives, helping to potentially prevent alert fatigue.
Example 4: Setting Up Processes for Responding to Alerts
Another way to help team members deal with alert fatigue is by setting up specific procedures and responses for handling alerts. If team members know how they must respond to an alert, it will take the decision about whether to respond out of their hands.
Having processes like these in place may have prevented one of the world’s largest data breaches. When Target Corp. suffered a security breach in 2014, leading to a significant loss of customer information and financial data, the company’s security software notified Target’s security team about the presence of malicious software on its network.
However, because of alert fatigue, the security team members chose to ignore the alert. The particular security alert did not stand out from any other of the dozens or hundreds of alerts the team was receiving, and the team failed to follow up on it later.
Had a series of procedures been in place that required the team to either respond immediately to the alert or to follow up on it later, a different outcome would’ve been possible.
Certainly, forcing team members to respond to every alert will not fix problems with alert fatigue. But taking the decision out of the team’s hands and giving them a system to follow every time will lead to better consistency in alert response and allow the team to immediately know whether an alert is urgent or not.
You may find with this policy in place that it will take hiring additional security team members to respond to all of the alerts. By having more employees working on security, individual team members will each need to respond to fewer alerts, which should help with alert fatigue.
How to Get Started With Preventing Alert Fatigue
The challenge of preventing alert fatigue sounds simple enough. Security teams simply must come up with a system that strikes the perfect balance between giving team members the alerts they need without overwhelming them.
Actually putting such a system into practice is a significant challenge, though. Here are some ideas for adjusting an alert system to combat alert fatigue.
Combining Alerts
Sometimes, a single major alert will generate several other alerts for minor issues.
A potential breach of the group’s passwords may generate an alert that all team members should start using two-factor authentication. But it also may generate multiple secondary alerts, such as for changing passwords, using maximum strength passwords, and strange network actions.
Rather than generating multiple alerts for this single event, tweaking the system to generate a single alert that encompasses the main alert and all of the offshoots from it may help with alert fatigue.
Change Responsibilities for Alerts
Rather than having one person responsible for all alerts all of the time, change up the responsibilities for responding to alerts weekly or perhaps even daily.
If team members only need to respond to alerts on certain days, they hopefully will have fewer problems with alert fatigue, knowing they will receive breaks on the other days.
Carry Proper Staffing to Handle Alerts
When a team doesn’t have enough personnel to handle all of the alerts, the chances of alert fatigue become greater. As team members feel overwhelmed with their normal duties, they may make a conscious choice to ignore an alarm, hoping that it’s a false alarm.
To fully implement an alert system for the security team, you may find that you need to hire more personnel to help with monitoring and responding to alerts.
Make Regular Tweaks to the System
No system to combat alert fatigue is going to be perfect the first time your organization deploys it. Tweaking the system to make it work more efficiently is an important part of deploying and using an alert system.
After the initial deployment, teams should monitor the system, determining which areas are generating too many alerts, leading to alert fatigue. The team should note areas where a useful number of alerts are occurring too, working toward using these success stories throughout the security plan.
Security teams should set up a regular time to review and adjust the system. For a new system, review times may need to occur monthly or quarterly. For an established system, holding reviews once or twice a year may be sufficient.
Implementing Machine Learning
Multiple technologies are available that help to sort alerts, reducing the number of false alarms and automatically handling some alerts without the need for human intervention.
With artificial intelligence and machine learning practices in place, the system will be able to learn the patterns of false alarms and of alarms that don’t require human intervention. Eventually, the system will stop generating alerts for these items, reducing the chances of developing alert fatigue.
Using these technologies gives the team members fewer alarms to deal with, letting them focus more on the most serious issues.
Actively Preventing Intrusions
Ultimately, the best way to avoid alert fatigue is to develop a security system that prevents intrusions and keeps the system as safe as possible. When the system has fewer reasons to generate alarms, the team will be less likely to develop alert fatigue problems.
Closely review the system setup on a regular basis, ensuring that it is taking advantage of the latest technologies and processes to operate as efficiently as possible.
