Limiting data being logged using Application Insights in Azure Functions

As you may know, Azure Functions have a preview of Application Insights integration enabled. This is another great addition to our serverless architecture since we don't have to add this dependency on our own - it's just there. However, there're some problems when it comes to handling the amount of data, which is being collected, especially when your're on an MSDN subscription.

Problem

When you enable Application Insights for your Function App, each function will start collecting different metrics(traces, errors, requests) at different scale. When you go to Azure Portal and access Data volume management tab in the Application Insights blade, you'll see, that there's one metric, which really exceeds our expectations(at least when it comes to the volume of the data traced):

As you can see, Message data takes 75% of the total amount of data collected

When you click on any bar, you'll access Data point volume tab - now we can understand, what kind of 'message' is really being logged:

Although chart says Message, data type related to this particular type of message is Trace

Configuring AI integration

Logging traces is perfectly fine, however we don't always want to do so(especially if you're on an MSDN subscription and don't want to be blocked). If you go to this page, you'll see a detailed information regarding both enabling and working with Application Insights. The most interesting part for us is the configuration section:

/
{
  "logger": {
    "categoryFilter": {
      "defaultLevel": "Information",
      "categoryLevels": {
        "Host.Results": "Error",
        "Function": "Error",
        "Host.Aggregator": "Information"
      }
    },
    "aggregator": {
      "batchSize": 1000,
      "flushTimeout": "00:00:30"
    }
  },
  "applicationInsights": {
    "sampling": {
      "isEnabled": true,
      "maxTelemetryItemsPerSecond" : 5
    }
  }
}

As you can see, we're able to set different levels for each category of data being logged. According to comments in this issue on GitHub, the easiest way to actually limit the data being logged is to set your configuration to the following:

/
{
  "logger": {
    "categoryFilter": {
      "defaultLevel": "Error",
      "categoryLevels": {
        "Host.Aggregator": "Information"
      }
    }
}

This way you should be able to avoid logging to much data or hitting your daily cap for Application Insights.

I strongly recommend you to play with AI integration in Azure Functions and provide feedback regarding possible features or enhancements. It's a great way to collaborate with a team responsible for a product and a chance to make it even better.

Proper deployment of your Function App using ARM template

In one of my projects I'm using ARM template to deploy both App Service plan and Function App to Azure. The important thing here is the fact, that I'm using a Free tier, so e.g. Always On property is not available for me. In this post I'll show you a small gotcha, which made my functions work improperly.

The structure

Currently the part of my project we're talking about looks like this:

  • App Service plan(Free)
  • Function App containing two functions - one is triggered every 5 minutes, fetches data from a FTP server and pushes it to a queue, the second one receives items from a queue and pushes them to a storage

Really simple, what could wrong here? As you probably know, Function Apps have two modes when it comes to choosing a hosting plan:

  • Consumption Plan(you pay for what you've used)
  • App Service Plan(predefined capacity allocation)

One important thing here is one of features according to Consumption Plan - with it you don't need Always On enabled for your Function App because each time a function is triggered, it is automatically awaken and processed. If you choose App Service Plan, you have to have Always On enabled - if not, your functions may work only occasionally. Since tier Free doesn't allow you to turn on this feature, using wrong consumption plan really breaks your setup.

The problem

When working on the mentioned projected during the last weekend I realized, that when I have a one day break from work, storage is missing items from that day. When I accessed functions to check whether everything is all right, missing items were immediately pushed to it. The same was true for Application Insights attached to my Function App - there was a big gap in the data collected. The conclusion was simple - there's something wrong with a hosting plan. I had to double-check my ARM template.

The template & solution

The part of the template responsible for provisioning my Function App looked like this:

/
{
  "name": "[parameters('liczniknetFunctionAppServicePlanName')]",
  "type": "Microsoft.Web/serverfarms",
  "location": "[resourceGroup().location]",
  "apiVersion": "2014-06-01",
  "dependsOn": [],
  "tags": {
	"displayName": "liczniknet-functionapp-serviceplan"
  },
  "properties": {
	"name": "[parameters('liczniknetFunctionAppServicePlanName')]",
	"workerSize": "[parameters('liczniknetFunctionAppServicePlanWorkerSize')]",
	"numberOfWorkers": 1
  }
}

For some reason I was completely sure, that the default mode for a Function App will be Consumption Plan. There's one small thing - when a Service App plan is being provisioned, it's not aware of resources, which will be deployed further. The solution was quite simple, I was missing two additional properties:

/
{
  "name": "[parameters('liczniknetFunctionAppServicePlanName')]",
  "type": "Microsoft.Web/serverfarms",
  "location": "[resourceGroup().location]",
  "apiVersion": "2014-06-01",
  "dependsOn": [],
  "tags": {
	"displayName": "liczniknet-functionapp-serviceplan"
  },
  "properties": {
	"name": "[parameters('liczniknetFunctionAppServicePlanName')]",
	"computeMode": "Dynamic",
	"sku": "Dynamic",
	"workerSize": "[parameters('liczniknetFunctionAppServicePlanWorkerSize')]",
	"numberOfWorkers": 1
  }
}

With this new setup my Function App is deployed with a proper Hosting Plan and I don't have to awake it manually.