The current state of play with Application Insights and Open Telemetry

Intro

Every developer knows that its important to somehow monitor what’s happening to their applications in the wild. Over the years there have been many attempts to standardise on an approach that makes sense in many different contexts. Distributed computing and the cloud just throws additional challenges into the mix. This is why a bunch of really bright folks in our industry (including Microsoft) got together to form the Open Telemetry project.

I spent considerable time in many different areas, and I’ll blog about these in future posts. The area I’ll cover today is the current state of Open Telemetry in .Net and Azure.

The current state of Open Telemetry for Application Insights (Azure Monitor)

Because we’ve invested heavily in using Application Insights in our applications, an obvious jumping off point for us would be to see if there was an Open Telemetry Exporter for Application Insights. Luckily, there is one here.

At the time of writing this is still in preview but sample code is provided and it was relatively straight-forward to see it up and running. After having spent a long time understanding the different types of telemetry that Open Telemetry talks about, it wasn’t immediately obvious how to find the raw data in the Application Insights logs. This is because Application Insights wasn’t developed with Open Telemetry in mind. Here’s a friendly mapping table for the confused (like me) :D.

Sampling

The Open Telemetry implementation seems to (currently) only support Fixed rate sampling. We use adaptive sampling and that gives us the ability to turn off sampling altogether for certain telemetry types if we wanted to. It also offers you to use your own post processing code to stop you emitting too much telemetry which seems very useful.

Code

If you’ve used Application Insights before you’re used to things pretty much working out of the box. Requests and dependencies just work. For the most part it’s the same using Open Telemetry. If you want to work with explicit spans or events you’ll need to pull in the System.Diagnostics namespace and work with Activities directly.

using (var activity = source.StartActivity("THIS IS A SPAN...))
{
    ... some really marvelous code
}

Are custom spans any better than the judicious use of log statements that give context to a trace? The code looks noisier.

Support for attributes and baggage is also available. You can read the open telemetry documentation for more context on how what they are.

Application Insights Experience

I won’t cover every area of Application Insights here, just the main areas most people are familiar with.

Application Map

Up until recently this didn’t show anything but it does now. Because spans are translated into dependencies in Application Insights, the application map will show your span with text as a dependency. I hope they fix this soon.

Live metrics

Unsupported (at the time of writing).

Transaction Search

This is fully supported and you will see a nice tree view of the entire transaction hierarchy. This is where spans show real merit as the text shows up nicely in the transaction tree. This allows you to get the full context of any transaction without having to expand the View all button.

Failures

Full support

Performance

Full support

Conclusion

The Open Telemetry feature for Azure Monitor (App Insights), while in active development, is still not at feature parity with the Application Insights SDK if you’re a .net development shop. You can keep abreast of the development here.

The first negative for me is the lack of support for adaptive sampling. Fixed rate sampling is a bit too coarse for our needs.

Then there is the question of code changes we would have to make to existing applications in order to make them work. While not that challenging, they are still an investment that doesn’t really offer any additional value for our customers today.

On a positive note, the Open Telemetry collector seems far more performant to using the SDK. Data can be seen much sooner in Application Insights. Will the days of the dreaded ‘n’ minute delay finally be near an end?

It’s all moving in a positive direction. I’d prefer to see the tooling mature a bit further, as well as ironing out some of the issues I’ve already talked about in this post. Given Microsoft’s investment in Open Telemetry, and their stated aim to move Azure Monitor to embrace the Open Telemetry specifications, I feel it is safe enough to wait for now.