Blog

If you’re interested in observability, it is very likely that you read Mat Duggan’s Why Clickhouse is Winning the Observability Wars or one of the follow ups and responses to it, for example Charity Majors’. Both are fantastic articles, and I recommend that you spend the time reading them front to back, give them your attention! And by that I also want to get one thing out of the way, that the title here might suggest: I do not disagree with them, I actually agree wholeheartedly with them on why ClickHouse is the current dominant backend in Observability. And they deserve that spot, as I know first hand from working with their product and knowing some of the great people working on it. ...

I’ve been in the observability space long enough, through OpenTelemetry, by seeing and doing dozens of vendor pitches and enough “AI-powered” demos to last a lifetime. So I have a clear idea about what a company needs to be successful in this area, and what would actually make me passionate about working for one. Read the full article on Bronto Blog

A while back I was exploring an open source project I was using, and interested in contributing to, as it provided capabilities I needed but missed some additional ones I wanted to have. Naturally, I was looking for a CONTRIBUTING.md. The file existed, but it only contained a note, that issues and PRs are accepted. While this is key information that belongs in this file, I was missing instructions on how to set up the repository, how to run tests, and what criteria I need to meet to increase my chances of having a PR accepted. ...

Throughout my jobs in the observability space, I created or contributed to various demo and sample applications, which often follow the same premise: there is a “normal state” in which the application is running, and with a trigger, it moves into a “deviated state”. For example, there is the placeOrder transaction on a webshop that performs just fine, and orders and money are flowing into our hypothetical e-commerce company. However, with the click of a button (or a CLI command), an issue is injected into the application, and the placeOrder transaction stops working as expected. Orders go down, money stops flowing, hypothetical customers get angry! ...

Modern reliability practice is excellent at making complex systems operable at the service level, but users experience reliability at the level of end-to-end transactions and flows. This post explores why that gap exists and what it means for how we measure and manage reliability. Read the full article on Causely Blog

“What’s your favorite telemetry signal” has been the last question of the Humans of OpenTelemetry series for the last few years. At KubeCon EU 2024 my answer to this was “profiling, because I think this is really closing a big gap that was missing in observability”. But, today, I found out that my answer has changed, and I am leaning more toward what Vijay Samuel gave as an answer: “I feel metrics are the most powerful signal!” ...

Slow SQL queries degrade user experience, cause cascading failures, and turn simple operations into production incidents. Instead of collecting more telemetry for its own sake, this guide shows how to turn OpenTelemetry database spans into span-derived metrics you can actually act on. Read the full article on Causely Blog

PagerDuty fires: CheckoutAPI burn rate (2m/1h). Grafana shows p99 going from ~120ms to ~900ms. Retries doubled. DB CPU is flat, but checkout pods are throttling and a downstream dependency’s error budget is evaporating. Read the full article on Causely Blog

People always say there are no stupid questions, and then you read the title of this post and you’re not so sure anymore. You start to doubt my sanity, or at least suspect that I’m a troll. However, as it is with most apparently stupid questions, there is something to learn from the answer if you explore it. To spare you from reading the rest of this, the short answer is “Yes, but…” and the long answer is more of a theoretical observation with some linguistic subtleties. So if you’re not interested in that, you can leave and do something fun, otherwise don’t say I didn’t warn you! ...

I did an experiment on splitting out a monolithic application into multiple “virtual services” in OpenTelemetry to have modules visualized independently on service maps. I am not sure if this is a good idea and something you should replicate in practice, since it might violate some best practices. However, I wanted to see how I can do it. Since (as far as I know) all otel backends are only able to provide such a map/graph visualization using service.name from the resource attributes, I tried out what happens if I create one TracerProvider per module with module-specific service.* attributes. ...