.png?width=114&height=26&name=Frame%20(16).png){kind=small}
The problem
Most people who have ever sat with Azure Service Bus as a transport layer in some kind of event-driven architecture have probably used Paolo Salvatori's WinForms project, Service Bus Explorer.
Personally, I have benefited a lot from Service Bus Explorer but I am genuinely surprised that it is the only tool available in its class to monitor and view queues in Service Bus, and the application lacks relevant features that I as a developer and system administrator need to do a good job.
The need to monitor and be able to alert when messages start to accumulate on a queue or DLQ has been recurring over the past six months. The specific case I have worked on has required the solution to be able to do the following:
- Monitor number of messages on deadletter queue (Azure Service Bus)
- Identify problems based on rules that can be configured, alert and then send e-mails
- Continuously update an Elasticsearch index with details about queues
- Visualize this Elasticsearch index on a Dashboard in Elasticsearch
To do this, we have developed a simple Azure Function that runs every five minutes, alerts if the configuration dictates and updates Elasticsearch with the right data. A few simple steps to achieve this solution.
Set up the Elasticsearch Index
The following curl creates an index in Elasticsearch named example-queue.
curl -X PUT https://{username}:{password}.{elasticsearch-instance}:9243/example-queue
Set up Azure Function
In the Azure portal, we set up an Azure Function container. We call it example-queue-monitor. When there is an Azure Function in place, we can take the opportunity to set it up with the right settings.
[
{
"name": "AlertRules:FiveMinutePercentageThreshold",
"value": "50",
"slotSetting": false
},
{
"name": "AlertRules:HighThreshold",
"value": "100",
"slotSetting": false
},
{
"name": "AlertRules:LowThreshold",
"value": "2",
"slotSetting": false
},
{
"name": "AlertRules:MediumThreshold",
"value": "10",
"slotSetting": false
},
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "",
"slotSetting": false
},
{
"name": "AzureWebJobsStorage",
"value": "",
"slotSetting": false
},
{
"name": "ElasticConnection",
"value": "https://{username}:{password}.{elasticsearch-instance}:9243",
"slotSetting": false
},
{
"name": "ElasticIndex",
"value": "example-queue",
"slotSetting": false
},
{
"name": "EmailConfig:Recipient",
"value": "where.to.send.the.alarm@example.com",
"slotSetting": false
},
{
"name": "EmailConfig:Sender",
"value": "valid.postmark.sender@example.com",
"slotSetting": false
},
{
"name": "Environment",
"value": "user for logging",
"slotSetting": false
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~2",
"slotSetting": false
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "dotnet",
"slotSetting": false
},
{
"name": "LogQueue",
"value": "elastic-log-index",
"slotSetting": false
},
{
"name": "PostMarkWarningTemplateId",
"value": "15046353",
"slotSetting": false
},
{
"name": "ServiceBusConnection",
"value": "Endpoint=sb://my- servicebus.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey={key}",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "",
"slotSetting": false
},
{
"name": "WEBSITE_RUN_FROM_PACKAGE",
"value": "",
"slotSetting": false
}
]
ServiceBusMonitorFunction
ServiceBusMonitorFunction is available on GitHub. We publish the function to our Azure Function and run it through once.
Elasticsearch
In addition to our new index, example-queue, we need an index pattern to be able to visualize the data in a meaningful way. We create it in the Elasticsearch dashboard. In our case, we can have an exact pattern which is also called example-queue.
I
n step 2, we can choose whether to use Created or Updated as the Time attribute in Elasticsearch, which is used for filtering. In our case we use Updated.
Elasticsearch Dashboard
There are many different ways to visualize data in Elasticsearch, but for this purpose we will settle for a simple pie chart.
So in the end we have a very simple monitoring with email sending when messages are built up on DLQ. Some things we know about is that the current solution sends an email every five minutes unless the deadletter queues are processed and emptied, that can be both good and bad depending on who receives the email ;)
Written by Marcus Nordquist