Custom Metrics to Azure Monitor - REST API Endpoint

Developers at Mystique Unicorn are looking for a way to track the performance of their application using custom metrics. Their application is creating the metrics in a JSON format

{
  "time": "2023-04-05T20:36:26",
  "data": {
    "baseData": {
      "metric": "QueueDepth",
      "namespace": "QueueProcessing",
      "dimNames": [
        "QueueName",
        "MessageType"
      ],
      "series": [
        {
          "dimValues": [
            "ImagesToProcess",
            "JPEG"
          ],
          "min": 2,
          "max": 17,
          "sum": 27,
          "count": 3
        }
      ]
    }
  }
}

They are looking for a central way to store and visualize them. Can you show them how the can get started?

🎯 Solution

We can use Azure Monitor Custom metrics¹ for our solution. These can be performance metrics or business/app specific. We have multiple ways of doing so, the most common ways are

1. Bootstrapping the application with Azure Application Insights SDK to send custom telemetry
2. Using Azure Monitor Agent²
3. Send them directly using the Azure Monitor REST API

In this blog, we are going to try out the REST API method. It gets us to understand the flow of things easily and with this knowledge in a later blog we will use the SDK.

1. 🧰 Prerequisites

   This demo, instructions, scripts and bicep template is designed to be run in westeurope. With few or no modifications you can try it out in other regions as well(Not covered here).

   • 🛠 Azure CLI Installed & Configured - Get help here
   • 🛠 Bicep Installed & Configured - Get help here
   • 🛠 VS Code & Bicep Extenstions - Get help here

2. ⚙️ Setting up the environment

   • Get the application code

     https://github.com/miztiik/custom-metrics-to-azure-monitor
     cd custom-metrics-to-azure-monitor

3. 🚀 Prepare the environment

   Let check you have Azure Cli working with

     # You should have azure cli preinstalled
     az account show

   You should see an output like this,

    {
      "environmentName": "AzureCloud",
      "homeTenantId": "16b30820b6d3",
      "id": "1ac6fdbff37cd9e3",
      "isDefault": true,
      "managedByTenants": [],
      "name": "YOUR-SUBS-NAME",
      "state": "Enabled",
      "tenantId": "16b30820b6d3",
      "user": {
        "name": "miztiik@",
        "type": "user"
      }
    }

4. 🚀 Deploying the application

   • Register an App to Azure AD - Doc Ref³

     • Create the client secret as well. Note down its value, as it is only shown during the creation time.

   • Create & Authorize a Service Principal to emit metrics (preferably a VM/FunctionApp)Doc Ref⁴

     • Follow the rest of the instructions

   • Get Auth Token:

     #Get TenantID
     az account show
     # or
     az account tenant list

     I have wrapped the instructions into a small bash script push_custom_metrics.sh. It will get the Auth Token from AAD, create a metric on JSON format with random values and make a POST request to the REST API to push the custom metric. The script will emit 10 metrics by default with 5 second interval between them. Adjust accordingly to your experimentation needs. Ofcourse update the Id, Secrets, ResourceID as needed

     #!/bin/bash
     set -x
     
     SLEEP_AT_WORK_SECS=5
     LOG_COUNT=10
     RES_LOCATION="westeurope"
     HOME_TENANT_ID=$(az account show --query homeTenantId -o tsv)
     APP_CLIENT_ID="50606203bc"
     APP_CLIENT_SECRET="x.J8Q~gc83"
     
     RESP=$(curl -X POST "https://login.microsoftonline.com/${HOME_TENANT_ID}/oauth2/token" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "grant_type=client_credentials" \
     --data-urlencode "client_id=${APP_CLIENT_ID}" \
     --data-urlencode "client_secret=${APP_CLIENT_SECRET}" \
     --data-urlencode "resource=https://monitor.azure.com")
     
     ACCESS_TOKEN=$(echo $RESP | jq -r '.access_token')
     echo $ACCESS_TOKEN
     VM_RES_ID="subscriptions/1e3/resourceGroups/Miztiik_Enterprises_custom_metrics_to_azure_monitor_002/providers/Microsoft.Compute/virtualMachines/m-web-srv-002"
     # VM_RES_ID=$(curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance/compute/resourceId?api-version=2021-05-01&format=text")
     ACCESS_TOKEN1="eyJ0eXAiOrWCAtgHzh6Jf0vTFA"
     
     
     for ((i=1; i<=LOG_COUNT; i++)); do
     
         # Set variable values
         CURR_TIME=$(date +"%Y-%m-%dT%H:%M:%S")
         TIME_TWO_HRS_AGO=$(date -d "-2 hours" +"%Y-%m-%dT%H:%M:%S") 
         METRIC="QueueDepth"
         NAMESPACE="QueueProcessing"
         QUEUE_NAME="ImagesToProcess"
         MESSAGE_TYPE="JPEG"
         MIN=$(shuf -i 1-10 -n 1)
         MAX=$(shuf -i 11-20 -n 1)
         SUM=$(shuf -i 21-30 -n 1)
         COUNT=$(shuf -i 1-5 -n 1)
     
         # Build JSON string
         JSON_DATA="{\"time\":\"$CURR_TIME\",\"data\":{\"baseData\":{\"metric\":\"$METRIC\",\"namespace\":\"$NAMESPACE\",\"dimNames\":[\"QueueName\",\"MessageType\"],\"series\":[{\"dimValues\":[\"$QUEUE_NAME\",\"$MESSAGE_TYPE\"],\"min\":$MIN,\"max\":$MAX,\"sum\":$SUM,\"count\":$COUNT}]}}}"
     
         echo "$JSON_DATA"
     
         curl --write-out %{http_code} -X POST "https://${RES_LOCATION}.monitoring.azure.com/${VM_RES_ID}/metrics" \
         -H 'Content-Type: application/json' \
         -H "Authorization: Bearer $ACCESS_TOKEN" \
         -d "$JSON_DATA"
     
         sleep $SLEEP_AT_WORK_SECS
     done
     

     Make a note of the response, particularly the access token

     • IMPORTANT - Do not forget to update the time parameter in the json. This should 30minutes behind UTC or 4minutes in the future. But always from UTC time.

5. 🔬 Testing the solution

   • Connect to the VM

     Chart the metric from Azure Monitor Portal

     

6. 📒 Conclusion

   Here we have demonstrated how to push custom metrics to the REST API.

7. 🧹 CleanUp

If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

• Resources created during Deploying The Application
• Any other custom resources, you have created for this demo

# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt

This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

👋 Buy me a coffee

Buy me a coffee ☕.

📚 References

1. Azure Docs: Azure Custom Metrics
2. Miztiik Blog: Send custom metrics from Vm to Azure Monitor using AMA
3. Azure Docs: Register App to get Auth Tokens from AAD
4. Azure Docs: Send custom metrics to the Azure Monitor metrics store by using a REST API

🏷️ Metadata

Level: 100