Splunk allows you to create either real time or scheduled alerts, and while real time alerts would seem logical they can be quite CPU intensive. In fact, some suggest limiting to 1 real time search per CPU core even though the default limits will be about 3 per core. You can see more on the limits.conf and how to change it here. Rather than creating real time alerts, you can set or change existing alerts to run on a schedule. In reality, most monitoring software works on a polling interval anyways so this is not far from what you are likely doing today with something like Nagios.
The process for creating or changing Splunk alerts from real time to scheduled is fairly straight forward. By default, however, the shortest time period Splunk provides is 1 hour. If you want to schedule Splunk alerts to run more frequently you will need to use the “Run on CRON scheduler” option. Cron schedule examples give me a bit of a headache, but since we only have to worry about a per minute time interval since Splunk provides other options here is a quick how to on how to set these up, or change them.
Changing a Splunk Alert from Real Time to Scheduled
- Log into your Splunk server
- Under Search and Reporting click on Alerts
- Find the Alert you wish to change and click Edit >> Edit Alert type and trigger
- Under Alert type click on Scheduled
- Change the time range to Run on CRON schedule, or one of the other options if those better suit your need
- The “earliest” text box should be a negative number that matches how often you will run the alert. For example if you want it to run every 5 minutes, set this to -5m
- In the “latest” box enter Now so that it will search logs between 5 minutes ago and run time
- In the CRON Expression box enter
*/5 * * * *
- You can find more information about alert schedules here
- Your alert window should look something like:
- Click Next and configure the Actions you want to enable, such as email subject, recipients and how to include information (inline, CSV or PDF) or even options such as running a script
- Click Save. Your alert will now run at the defined schedule.
While alerts can be swell, they will have an impact on your server and, if you have to many alerts, you might not actually receive any of them, which would be bad. Scheduling alerts is an easy way to make sure your Splunk server is not resource constrained.