Learn how Log Search Alerts Work in Azure

Log Search Alerts in Azure Monitor work by allowing you to monitor and be notified of specific events or conditions based on log data generated by your Azure resources. These alerts are created by running Kusto Query Language (KQL) queries on logs stored in Azure Log Analytics workspaces and other log sources, such as Activity Logs, Azure Diagnostics Logs, and Azure Security Center Logs.

How Log Search Alerts Work in Azure

Here’s a step-by-step breakdown of how log search alerts are triggered:

Log Data Collection

Azure resources generate log data that is stored in Azure Log Analytics. This log data can be generated by various Azure services like:

1. Activity logs (for management operations)

2. Azure Monitor logs (for platform metrics, diagnostics, etc.)

3. Application Insights logs (for application performance)

4. Custom logs from VMs or containers

Logs are collected and sent to Log Analytics workspaces.

Create a Log Search Query

To create a log search alert, you first need to define a query that filters the relevant log data using Kusto Query Language (KQL). KQL allows you to perform advanced searches on logs, filtering based on specific patterns, error codes, timestamps, or custom log data.

For example:

To find failed login attempts:

xxxxxxxxxx
3
1
SecurityEvent
2
| where EventID == 4625
3
| where TimeGenerated > ago(1h)

To search for high CPU usage in a virtual machine:

xxxxxxxxxx
3
1
Perf
2
| where CounterName == "Processor Time" and ObjectName == "Processor" and InstanceName == "_Total"
3
| where TimeGenerated > ago(5m) and CounterValue > 90

Define an Alert Condition

Once you have your KQL query, you can define a condition that will trigger the alert when the query results meet certain criteria. For example:

1. Threshold-based conditions: If the query returns a certain number of log entries within a specified time period (e.g., more than 10 failed login attempts in 5 minutes).

2. No results conditions: Alert when no records match a query in a specified time window.

3. Aggregation: Alerts can also be set based on the aggregation of data (e.g., the total number of errors across multiple services).

Set the Alert Triggering Frequency

You need to define how often Azure should evaluate the log search query. This is done by setting an evaluation frequency (e.g., every 5 minutes, every 1 hour). The alert will run the query periodically at the set frequency.

Define the Alert Action

Once the alert condition is met, Azure will trigger the alert. You can configure Action Groups to determine what happens next. Action Groups define the notifications or actions to be taken when an alert is triggered. Actions could include:

1. Sending an email or SMS notification.

2. Triggering a webhook.

3. Calling an Azure Function.

4. Creating a ticket in a service management platform.

You can use an existing Action Group or create a new one while setting up the alert.

Create the Alert Rule

Once the query and actions are set, you can create the Log Search Alert Rule. This rule defines the log search query, alert condition, and the action to take when the alert is triggered.

The alert rule will be continuously evaluated based on the configured frequency, and if the condition is met, the alert will be triggered.

Example of Log Search Alert Workflow

Here’s how a log search alert works step by step:

1. Log data is generated by Azure services or your application and sent to Log Analytics workspace.

   • You create a KQL query to search the log data for specific events (e.g., error messages, security events, or performance issues).

   • You define the alert condition (e.g., more than 10 failed login attempts in the last 5 minutes).

2. Azure Monitor evaluates the log data at the specified frequency (e.g., every 5 minutes).

   • If the query matches the defined condition, the alert is triggered.

3. Azure Monitor then executes the Action Group, sending out notifications or taking other actions (e.g., invoking a webhook, calling an Azure function).

Log Search Alert Use Cases

1. Security Monitoring: Track specific events such as failed login attempts, suspicious user activity, or unauthorized access.

2. Performance Monitoring: Monitor application or server logs for performance degradation (e.g., CPU spikes, memory issues, or application crashes).

3. Operational Monitoring: Detect critical system errors or warnings that could indicate problems with Azure services or your application.

4. Compliance: Track changes to configurations or policies, helping to monitor for security and regulatory compliance violations.

5. Custom Applications: If you have custom logging, you can track business-specific events, like transaction failures or unusual user behaviors.

Benefits of Log Search Alerts

1. Flexibility: You can monitor any log data that is ingested into Log Analytics, including custom logs from your applications.

2. Advanced Querying: Using KQL, you can write complex queries to identify specific patterns and conditions in your logs.

3. Proactive Monitoring: Log search alerts enable you to detect issues early by setting up custom queries and conditions.

4. Scalability: These alerts can be scaled to monitor multiple Azure resources, applications, and services.

Key Considerations

1. Query Performance: Complex queries might require more resources and take longer to evaluate. Ensure that your queries are optimized for performance.

2. Cost: The frequency of log query evaluations and the volume of logs being queried can impact Azure Monitor costs, especially when querying large amounts of data.

3. Alert Noise: To avoid excessive alerts, carefully set thresholds and conditions to ensure alerts are meaningful and actionable.

Summary

By setting up log search alerts, you can monitor your Azure resources and applications in a more detailed and customized way, responding quickly to potential issues based on log data.