Learn about Azure Log Analytics with detailed demonstration

Azure Log Analytics is a powerful tool within Azure Monitor that helps you collect, analyze, and act on log data from various Azure resources. With Log Analytics, you can query logs, monitor metrics, detect issues, and even automate responses based on log data. It is based on the Log Analytics Workspace, which is a centralized repository where you can store, search, and analyze log data using Kusto Query Language (KQL).

Let’s go step-by-step through the process of using Azure Log Analytics.

Step 1: Accessing Azure Log Analytics

1. Login to Azure Portal:

   • Sign in with your Azure account credentials.

2. Navigate to Azure Monitor:

   • In the left-hand side menu, select Monitor. Alternatively, use the search bar to type "Monitor" and select it.

3. Access Log Analytics:

   • Under General, click on Logs to open Log Analytics.

4. Select or Create a Log Analytics Workspace:

   • You will be asked to either select an existing Log Analytics Workspace or create a new one.

   • To create a new workspace:

     • Click + Create new.

     • Choose a Subscription and Resource Group.

     • Provide a name for the workspace.

     • Select the Region where the workspace will be located (pick the region closest to your resources).

     • Click Create to create the workspace.

Step 2: Querying Log Data with Kusto Query Language (KQL)

1. Select the Log Analytics Workspace:

   • After creating or selecting your workspace, click on the workspace to open its Logs blade.

2. Run a Sample Query:

   • In the Logs section, you will see a Query Editor where you can run Kusto Query Language (KQL) queries. This is used to search and analyze logs in the workspace.

   • For example, to search for activity logs:

   xxxxxxxxxx
   4
   1
   AzureActivity
   2
   | where TimeGenerated > ago(1d)
   3
   | where OperationName == "Create Resource"
   4
   | project TimeGenerated, ResourceGroup, OperationName, Resource
   

   • This query will return all the resource creation events that occurred in the last 24 hours.

3. Explaining the Query:

   • AzureActivity: This is the table containing activity logs for resource management operations (create, delete, update, etc.).

   • where TimeGenerated > ago(1d): Filters the logs to show entries from the last 1 day.

   • where OperationName == "Create Resource": Filters the logs to show only "Create Resource" events.

   • project: This operator is used to select specific columns of data for display (e.g., TimeGenerated, ResourceGroup, OperationName, Resource).

4. Run the Query:

   • Click Run to execute the query. The results will be displayed below the query editor.

Step 3: Create an Alert Based on Log Analytics Query

1. Create an Alert Rule:

   • After running a query and ensuring that it returns the desired results, you can create an alert based on this query.

   • Click on New alert rule at the top of the query results.

2. Define the Scope:

   • The scope will be automatically set to your Log Analytics Workspace. If you have multiple workspaces, select the appropriate one.

3. Set the Condition:

   • The condition will be set based on your query. Choose an alert condition like "Count" or "Threshold":

   • For example, you might want to be alerted if there are more than 5 resource creation events within the last 5 minutes.

   Example:

   xxxxxxxxxx
   5
   1
   AzureActivity
   2
   | where TimeGenerated > ago(5m)
   3
   | where OperationName == "Create Resource"
   4
   | summarize count() by OperationName
   5
   | where count_ > 5
   

   • In this query, the "summarize" function aggregates events, and the "where count_ > 5" condition ensures that if more than 5 resource creation events occur within the last 5 minutes, the alert is triggered.

4. Define the Action:

   Under the Action section, select or create an Action Group. You can configure actions such as:

   • Email notifications to a specific address.

   • Webhook to trigger a custom automation.

   • Azure Logic Apps to automate workflows.

5. Set Alert Details:

   • Provide a name for the alert rule (e.g., "High Resource Creation Alert").

   • Choose the severity level for the alert (e.g., Severity 2 for warning).

   • Add tags if needed for better organization.

6. Review and Create:

   • After reviewing your settings, click Create to finalize the alert rule.

Step 4: Analyzing and Filtering Logs

1. Using Filters to Refine Data:

   You can further refine your log queries using various KQL operators, such as:

   • project: Select specific columns to display.

   • summarize: Group and aggregate data (e.g., count, average).

   • top: Display the top N results based on a particular field (e.g., top 10 events by severity).

   • extend: Add new columns derived from existing data (e.g., convert time stamps to a different format).

   Example query to summarize events by operation name:

   xxxxxxxxxx
   3
   1
   AzureActivity
   2
   | summarize count() by OperationName
   3
   | top 10 by count_
   

2. Time Filtering:

   • Use the ago function to filter data based on time ranges. For example:

   • | where TimeGenerated > ago(1h) to get logs from the past hour.

   • | where TimeGenerated between(datetime(2024-12-01) .. datetime(2024-12-02)) to filter logs between specific dates.

3. Analyzing Custom Logs:

   • If your resources or applications are sending custom logs, you can also query these logs by specifying the appropriate log table (e.g., CustomLog, AppLogs).

Step 5: Visualizing Log Data

1. Pin Queries to Dashboard:

   • After running a query, you can pin the results to an Azure Dashboard for easy access and visualization.

   • Click on the Pin to Dashboard button in the query result pane.

   • Choose a dashboard or create a new one to visualize the query results in real-time.

2. Create Custom Dashboards:

   • You can create custom dashboards that aggregate multiple log queries, charts, and metrics to provide a consolidated view of your resource's performance and health.

   • Go to Dashboard > + New Dashboard, and then add your Log Analytics query results or metric charts.

Step 6: Using KQL for Advanced Analysis

1. Joining Tables: One of the powerful features of KQL is the ability to join multiple tables to correlate events. For example, you can join AzureActivity logs with SecurityEvent logs to detect failed login attempts for a specific resource.

xxxxxxxxxx
4
1
AzureActivity
2
| join kind=inner (SecurityEvent) on $left.Resource == $right.Resource
3
| where TimeGenerated > ago(1d)
4
| project TimeGenerated, Resource, OperationName, EventID

2. Using Functions: KQL allows you to define functions for reusability. For example:

xxxxxxxxxx
5
1
.create function GetErrorLogs() {
2
   AzureActivity
3
   | where Level == "Error"
4
   | summarize count() by Resource, OperationName
5
}

After defining a function, you can call it in queries to reuse the logic.

Step 7: Monitoring and Managing Logs

1. Log Search History:

   • Log search history helps track and manage previously run queries. You can access this under "Query History" and review the logs that were analyzed in the past.

2. Retention and Data Management:

   • You can set data retention policies for your Log Analytics workspace. Logs can be retained for a maximum of 2 years, after which older data is automatically purged.

   • To configure retention, navigate to your Log Analytics Workspace > Usage and estimated costs > Data Retention.

Step 8: Automating Responses to Log Analytics Data

1. Using Azure Automation:

   • You can use Azure Automation to automate responses based on alerts triggered by log queries. For example, a query might detect a service failure, and a runbook could automatically restart the affected service.

2. Azure Logic Apps:

   • Create workflows in Azure Logic Apps to automate incident management, send Slack messages, or create tickets in a service desk system.

Summary

In this detailed demonstration, we explored how to use Azure Log Analytics to collect, query, and analyze log data from your Azure resources. We also walked through how to create alerts based on Kusto Query Language (KQL), visualize log data on dashboards, and automate responses based on log queries. By effectively using Azure Log Analytics, you can enhance your monitoring, troubleshooting, and automation strategies within Azure.