Blog Detail

Observing .NET microservices with OpenTelemetry - logs, traces and metrics

To explore these observability topics, we need a sample application. As usual, we want something complex enough to see things in action, but simple enough to not be a distraction from what we’re trying to learn.

So what we have is an application composed of two services:

• An HTTP API that receives POST requests, then pushes messages to a Kafka broker
• A background service that processes messages from Kafka, then writes something in a PostgreSQL database

Aside: about the choice of tools

Note that I don’t have super strong opinion about these tools, so, as usual, I take advantage of doing these experiments to try out technologies I didn’t use yet.

Going in, I had in mind using as much OpenTelemetry stuff as possible (as mentioned earlier), as well as Prometheus and Grafana, which seem like a sort of de facto standard in each one’s typical use cases, so I wanted to try them out.

As for Grafana Loki and Tempo, since I had to pick something, I thought I might as well go with something that integrated easily with the other already made choices. Besides that, and an overarching theme for all choices, I also wanted something that I could setup easily and run locally, so the sample code is as self contained as possible.

Logs

Regarding logging, we can get it out of the way rather quickly. Nothing changes in regards to using logging, we can continue to use Microsoft.Extensions.Logging.ILogger as usual, we just need to change a bit the configuration.

Firstly, to be able to export the logs to an OpenTelemetry compliant target, we need the OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs package. Then, we need to, first configure logging, followed by the export itself.

The following code, shows the logging configuration for the Worker service.