Discovering Schedules and Dependencies from Mongo Change Streams
Many systems already know a lot about themselves — you just have to listen. MongoDB change streams (CDC) emit a continuous feed of inserts, updates, and deletes. With a little routing into a fast analytical database like ClickHouse, you can let the system “discover itself”: jobs, runs, schedules, dependencies, and even the fingerprints of human intervention. 1. Capture the Raw Feed First, set up a connector: MongoDB → Kafka → ClickHouse In ClickHouse, land the JSON envelopes losslessly: ...