In this episode, host Amey Ambade sits with Eric Tschetter, co-founder of Apache Druid and Chief Architect at Imply, to dissect the critical move toward Decoupling Observability. To begin, they define three pillars—logs, metrics, and traces—and consider why the rise of microservices has made traditional, tightly coupled stacks a major source of pain. Such coupled systems can lead to issues such as vendor lock-in, prohibitive scaling costs, and operational complexity.
Drawing parallels to the Business Intelligence world’s separation, Tschetter presents an architectural solution with four distinct layers: Ingest/Route, Data Storage, Query/Compute, and Visualization. This framework aims to provide flexibility to combat the limitations of monolithic observability tools. The conversation moves into the practical challenges and significant benefits of this decoupled model, focusing heavily on data portability and the role of technologies such as OpenTelemetry in standardizing schemas so that data can flow freely between multiple back-ends. A significant portion of the discussion is dedicated to the Query/Compute layer, specifically how Apache Druid addresses the unique demands of real-time analytics on observability data, including indexing strategies and unifying results across hot and cold storage. They also delve into operational survival, covering critical topics like smart sampling to preserve high-value signals, best practices for buffering and backpressure, and the governance models required for multiple teams to safely access the same data lake.
The episode concludes with an honest look at the complexity trade-offs and a roadmap for organizations considering a migration from a coupled vendor stack.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
-
- SE Radio 556: Alex Boten on OpenTelemetry — telemetry interoperability, collectors, and the OpenTelemetry project
- SE Radio 591: Yechezkel Rabinovich on Kubernetes Observability — three pillars of observability, eBPF, and observability costs
- SE Radio 455: Jamie Riedesel on Software Telemetry — foundational concepts of tracing, logging, and monitoring infrastructure
- SE Radio 534: Andy Dang on AI/ML Observability — observability for ML applications, data drift, and production failures
- SE Radio 610: Phillip Carter on Observability for LLMs — observability-driven development and debugging LLMs