Blog

Throughout my jobs in the observability space, I created or contributed to various demo and sample applications, which often follow the same premise: there is a “normal state” in which the application is running, and with a trigger, it moves into a “deviated state”. For example, there is the placeOrder transaction on a webshop that performs just fine, and orders and money are flowing into our hypothetical e-commerce company. However, with the click of a button (or a CLI command), an issue is injected into the application, and the placeOrder transaction stops working as expected. Orders go down, money stops flowing, hypothetical customers get angry! ...

“What’s your favorite telemetry signal” has been the last question of the Humans of OpenTelemetry series for the last few years. At KubeCon EU 2024 my answer to this was “profiling, because I think this is really closing a big gap that was missing in observability”. But, today, I found out that my answer has changed, and I am leaning more toward what Vijay Samuel gave as an answer: “I feel metrics are the most powerful signal!” ...

People always say there are no stupid questions, and then you read the title of this post and you’re not so sure anymore. You start to doubt my sanity, or at least suspect that I’m a troll. However, as it is with most apparently stupid questions, there is something to learn from the answer if you explore it. To spare you from reading the rest of this, the short answer is “Yes, but…” and the long answer is more of a theoretical observation with some linguistic subtleties. So if you’re not interested in that, you can leave and do something fun, otherwise don’t say I didn’t warn you! ...

I did an experiment on splitting out a monolithic application into multiple “virtual services” in OpenTelemetry to have modules visualized independently on service maps. I am not sure if this is a good idea and something you should replicate in practice, since it might violate some best practices. However, I wanted to see how I can do it. Since (as far as I know) all otel backends are only able to provide such a map/graph visualization using service.name from the resource attributes, I tried out what happens if I create one TracerProvider per module with module-specific service.* attributes. ...

When you’re heads-down in your own area of expertise, it’s easy to forget that what’s obvious to you might not be to others. As you might have seen in previous posts, I learned that for me using pen and paper from time to time helps uncover unknown knowns in my head. Last time, it was why the three pillars need to go. This time, it’s context propagation, and its surprising relationship to metrics. ...

I just read another post introducing traces, metrics, and logs using that analogy, which reminded me to re-share that excellent piece by Ted Young on The New Stack from a few years ago: Modern Observability Is a Single Braid of Data Ted argued the pillars are no longer load‑bearing and suggests a better framing: the “Single Braid of Data”. So let’s wheel the pillars into the museum, rope off the exhibit, and hang a small plaque: “Historic framing.” As we do with once‑cherished pillars that are no longer load‑bearing. ...

Most developers use automatic instrumentation without knowing how it actually works. This post breaks down the key techniques behind it—not to build your own, but to understand what’s really happening when things “just work.” Read the full articles on: Causely Blog, OpenTelemetry Blog

I have a confession to make: I struggled for a long time with the statement: “Observability helps you uncover unknown unknowns.” It never quite sat right with me. If you can uncover an “unknown unknown” using Observability, wasn’t it always, in some sense, a known unknown? Or perhaps even an unknown known you just hadn’t noticed? That logical loop kept bothering me. Recently, I realized where my understanding failed, and thought this might be worth sharing: ...