Extend the MongoDB → Kafka → ClickHouse pipeline with ServiceNow ticket data to provide early-warning signals for incidents, helping on-call engineers see problems before tickets are created.
ClickHouse schema definitions and example queries for the TicketSoon pilot, integrating MongoDB CDC, system events, and ServiceNow tickets into a unified event store.
Many systems already know a lot about themselves — you just have to listen. MongoDB change streams (CDC) emit a continuous feed of inserts, updates, and deletes. With a little routing into a fast analytical database like ClickHouse, you can let the system “discover itself”: jobs, runs, schedules, dependencies, and even the fingerprints of human intervention. 1. Capture the Raw Feed First, set up a connector: MongoDB → Kafka → ClickHouse In ClickHouse, land the JSON envelopes losslessly: ...
When you already have systems tracking their own state in MongoDB, you can turn that into a real-time stream of structured events without rewriting application logic. This approach captures every meaningful change from Mongo, tags it with relevant metadata, and makes it instantly queryable in ClickHouse — all through a generic, reusable pattern. The idea: One fixed event envelope for all sources Dynamic tags/attributes defined in config files No code changes when onboarding new collections 1. The Fixed Event Envelope Every CDC message has the same top-level structure, no matter what source system or collection it came from: ...
Once your MongoDB change streams are flowing through Kafka and landing in ClickHouse, you’ve got a live, queryable event history for every state change in your systems. The obvious next step: start using it immediately — even before you build full-blown dashboards or machine learning models. 1. Detect Missing or Late Events One of the fastest wins is catching when something doesn’t happen. If you know a collection normally sees certain events every day, you can query for absences: ...
This guide shows how to wire MongoDB Atlas → Amazon MSK (Kafka) → ClickHouse on AWS so that change events from existing MongoDB apps are captured in a fast, queryable store. You’ll get: A lossless CDC stream of MongoDB changes into Kafka An optional config‑driven normalizer to add tags/attributes A ClickHouse sink for sub‑second queries and analytics Security and cost controls that work in a typical enterprise VPC 0) Architecture at a Glance MongoDB Atlas (Change Streams) │ (Atlas Kafka Source Connector) ▼ Amazon MSK (Kafka) ──► [Optional] Normalizer (Kafka consumer) │ │ │ └─ emits normalized events ▼ ClickHouse on AWS ◄─────────────┘ (Kafka Engine table or small consumer) Why this shape? Atlas produces change events; MSK is your durable bus; ClickHouse gives you fast, tag‑rich queries and easy rollups. ...