The dashboard nobody noticed was broken

A regular day in the life of a data engineer.

I open Slack. One of the alert channels for Kafka pipelines is highlighted. First thoughts? I hope it’s someone else's pipeline.

There is a message from one of our SRE engineers. “This Kafka producer is failing. Who can take a look?”.

Ahh gosh. That’s mine. I must take a look.

The pipeline has been failing for two days. I thought we had all the necessary a…