How to reduce Application Insights telemetry data amount

hosting
,
1

There is the following problem with “overlogging” in Application Insights: VM or AppService resources drain. It’s very specific to your system, depending on modules, types of loads, etc., but too much telemetry can really slow your system.
You should be aware if you see something, like:

  1. Number of threads to service non-proportionally high for your system load (you have a couple or triple clients per second, but thousands of threads);
  2. You see CPU burning;
  3. If you have a dump, you discovered a lot of AI-specific active calls.

The solution is to reduce the amount of telemetry data passing by your system to Application Insights.
The platform, starting from v3.32, contains an ability to tune Application Insights telemetry by:

  1. Setting up standard telemetry sampling processors;
  2. Additional SQL-dependency ignoring processor.

Standard telemetry sampling processors

There are two standard sampling processors in Application Insights: adaptive and fixed. To learn the difference and detailed settings info take the topic Sampling in Application Insights.
Unfortunately, ApplicationInsights.config not supported with ASP.NET Core, but we implemented equivalent settings in the platform.
Look at the section VirtoCommerce:ApplicationInsights:SamplingOptions in appsettings.json:

        "SamplingOptions": {
            "Processor": "Adaptive",
            "Adaptive": {
                "MaxTelemetryItemsPerSecond": "5",
                "InitialSamplingPercentage": "100",
                "MinSamplingPercentage": "0.1",
                "MaxSamplingPercentage": "100",
                "EvaluationInterval": "00:00:15",
                "SamplingPercentageDecreaseTimeout": "00:02:00",
                "SamplingPercentageIncreaseTimeout": "00:15:00",
                "MovingAverageRatio": "0.25"
            },
            "Fixed": {
                "SamplingPercentage": 100
            },
            "IncludedTypes": "Dependency;Event;Exception;PageView;Request;Trace",
            "ExcludedTypes": ""
        }

You can choose the sampling processor in the Processor key between Adaptive and Fixed and tune each of them under sections Adaptive and Fixed respectively.

SQL-dependency ignoring processor

We implemented this additional processor to have an ability to ignore SQL-related dependencies, containing specified substrings. If one of the strings found in the dependency SQL statement it will be not sampled by Application Insights.
To enable SQL telemetry in dependencies set the value for key VirtoCommerce:ApplicationInsights:EnableSqlCommandTextInstrumentation to true. To setup the set of ignoring substrings, change the array value under key VirtoCommerce:ApplicationInsights:IgnoreSqlTelemetryOptions:QueryIgnoreSubstrings.
Full example of VirtoCommerce:ApplicationInsights section:

    "ApplicationInsights": {
        "SamplingOptions": {
            "Processor": "Adaptive",
            "Adaptive": {
                "MaxTelemetryItemsPerSecond": "5",
                "InitialSamplingPercentage": "100",
                "MinSamplingPercentage": "0.1",
                "MaxSamplingPercentage": "100",
                "EvaluationInterval": "00:00:15",
                "SamplingPercentageDecreaseTimeout": "00:02:00",
                "SamplingPercentageIncreaseTimeout": "00:15:00",
                "MovingAverageRatio": "0.25"
            },
            "Fixed": {
                "SamplingPercentage": 100
            },
            "IncludedTypes": "Dependency;Event;Exception;PageView;Request;Trace",
            "ExcludedTypes": ""
        },
        "EnableSqlCommandTextInstrumentation": true,
        "IgnoreSqlTelemetryOptions": {
            "QueryIgnoreSubstrings": [ "[HangFire].", "sp_getapplock", "sp_releaseapplock" ]
        }
    }

Upd.: Also, the platform, starting from v3.34, can configure every available standard Application Insights setting, thru platform appsettings.json.

2 Likes