Logs of Duty: The real-time pipeline

Because waiting 5 minutes for data is for losers.

Rule #1: Trust your Product Owner (or not). If the product team tells you they need real-time data, they probably do. Until you hear about that “small” requirement involving joining five data streams together and a CSV from 2020. Save your soul early. Ask questions.

Rule #2: Real-time means Kafka. Forget everything else you know, just use Kafka for every…