Articles

Karma now ingests, normalizes, and routes events from any CDC-like source into a shared ledger, optional graph, and an action loop — setting the stage for learned expectations and autonomous intervention.

In Karma, every action is just another event. This post explains the pattern for turning anomalies and rules into commands, tracking their execution, and feeding the results back into the same event pipeline.

Karma uses a single normalized event stream to feed both a ClickHouse ledger and an optional graph database — but through separate pipelines for flexibility, scalability, and clarity.

When you already have systems tracking their own state in MongoDB, you can turn that into a real-time stream of structured events without rewriting application logic. This approach captures every meaningful change from Mongo, tags it with relevant metadata, and makes it instantly queryable in ClickHouse — all through a generic, reusable pattern. The idea: One fixed event envelope for all sources Dynamic tags/attributes defined in config files No code changes when onboarding new collections 1. The Fixed Event Envelope Every CDC message has the same top-level structure, no matter what source system or collection it came from: ...

Once your MongoDB change streams are flowing through Kafka and landing in ClickHouse, you’ve got a live, queryable event history for every state change in your systems. The obvious next step: start using it immediately — even before you build full-blown dashboards or machine learning models. 1. Detect Missing or Late Events One of the fastest wins is catching when something doesn’t happen. If you know a collection normally sees certain events every day, you can query for absences: ...

This guide shows how to wire MongoDB Atlas → Amazon MSK (Kafka) → ClickHouse on AWS so that change events from existing MongoDB apps are captured in a fast, queryable store. You’ll get: A lossless CDC stream of MongoDB changes into Kafka An optional config‑driven normalizer to add tags/attributes A ClickHouse sink for sub‑second queries and analytics Security and cost controls that work in a typical enterprise VPC 0) Architecture at a Glance MongoDB Atlas (Change Streams) │ (Atlas Kafka Source Connector) ▼ Amazon MSK (Kafka) ──► [Optional] Normalizer (Kafka consumer) │ │ │ └─ emits normalized events ▼ ClickHouse on AWS ◄─────────────┘ (Kafka Engine table or small consumer) Why this shape? Atlas produces change events; MSK is your durable bus; ClickHouse gives you fast, tag‑rich queries and easy rollups. ...

Abstract This document outlines a foundational perspective for a possible future discipline of runtime epistemology — the study of how infrastructure systems can quantify their own state of divergence from intended behavior. It proposes that Shannon entropy offers a mathematically principled basis for measuring runtime drift in live systems, forming the core of a design pattern suitable for both operational reliability and machine-driven reasoning. Introduction Contemporary infrastructure systems are increasingly dynamic, distributed, and subject to change. While observability tools have improved, systems still rely on humans to reconcile what is happening with what was supposed to happen. This epistemic gap — the difference between actual and intended behavior — remains largely qualitative, ad hoc, and unmeasured. ...

Wire HTTP routes to Lambda functions using OpenAPI, Parameter Store, and Terraform — no hardcoded ARNs, no manual setup.

I’ve published a full working demo of a modern analytics stack: dbt transformations modeled with raw → staging → marts BigQuery as the cloud data warehouse Tableau dashboard visualizing business metrics Fully modular, automated, and environment-agnostic setup Explore the project: Analytics Repo on GitHub View Dashboard on Tableau Public Everything is built to be portable, extensible, and reproducible — designed both for demonstration and as a real starting point for more complex pipelines. ...

A high-level view of how Adage handles configuration-driven deployments Terraform wasn’t designed for object-oriented programming. It’s declarative, static, and prefers duplication over abstraction. But infrastructure doesn’t have to be. With the right separation of concerns, you can make your infrastructure modular, composable, and runtime-aware. You can design a system where components behave more like objects: isolated, testable, and able to change behavior based on identity and configuration. ...