OTel Isn't Going Well (And I Made A Spreadsheet About It)
36 points by pondidum
36 points by pondidum
Thanks for sharing such great insights into the OTel ecosystem. There really is a heroic effort being made by all the maintainers involved. I particularly agree with this section about expectations on the community and thanks for highlighting it!
In general with OTel though, I've never fully gotten on board with it. The promises have been enormous for years, but every major touchpoint I have had with it have been extremely painful. One major exception to this however was how easy it was to point my collectors at a new vendor to try out their service (Grafana -> Axiom). The one promise they have really fulfilled on has been their vendor agnosticism (as also pointed out in the article)!
I think 9 times out of 10, people just want logs. It's unfortunate that over the past years, logs seem to have really fallen out of favour within the observability community! Of course there is a need for other types of insights too, but in most cases, accessible logs are far more enabling than whatever you're getting from your metrics, spans, profiles, etc.
Spans really enhance logs quite a bit!
Tracing is this thing I want to love but really struggled with in reality. If you want to have full traces for rare errors, which is where they would be most useful, it seems to me you can't use sampling: you have to capture every span and every trace and at least keep them until the trace is done. And when is it done? Combine this with a system that produces 2TB of spans per hour, how do you actually deploy a solution for this?
For my homelab I'm currently setting up a newer stack since I haven't touched OTel in a few years, to figure out what I realistically can do, how can can capture rare failures with a trace of a distributed system. My load there will be seriously lower than that work, will be interesting to see how much compute I will spend on observability as compared to the applications themselves.
I’ve never done it at work but if I recall correctly you can do tail sampling in order to keep/filter out a trace when it’s done and not at the span level.
I imagine that you likely don’t have hour long traces so it should not require to hold TB of data to make it work.
But how do you know a trace is done, if you set a deadline and you get a span refering a "done" span, what should be done?
Anyway tail sampling sounds like the solution I needed at that job. Long left so don't know what happened.
Good question, I guess that you either drop it or make it appear as some kind of orphan span.
I hope to have time to implement it at work this year in order to know for real.
I build an ActivityPub framework, and I went with putting OpenTelemetry support (spans, metrics, span events) directly into the framework rather than waiting on a contrib package. It's worked out well enough for me that I've started to wonder whether that's generally the better default for frameworks, though I'm aware I'm generalizing from one project.
Part of my hesitation about contrib is that a contrib maintainer doesn't really own the framework's semantics. They have to guess what deserves a span versus an event versus a metric from the outside, and the names and attributes they land on tend to be generic rather than shaped by the actual domain. There's also a more mundane problem: in a lot of dynamic languages, contrib instrumentation ends up monkeypatching the framework's internals to get at anything useful, and that tends to be fragile, breaking quietly across minor version bumps in a way the framework's own maintainers wouldn't necessarily notice.
None of this is free, to be clear. Owning the instrumentation means the core package now carries something closer to an observability team's workload. I just suspect that's a cost worth paying over hoping someone picks up your project's semantics as a side project in contrib, though I could see this looking different for frameworks with more churn or a smaller maintainer base than mine.
I’m not super clear about what a contrib package is, but could the maintainer publish their “semantics” such that someone else can hook into them with an implementation? And if that’s what OTel already is (hooks for someone else’s implementation), then what’s the issue/cost?
I warmly recommend to implement your own instrumentation for your language of choice.
At work I did it for our Python/Django stack because looking at the one available on shelf did not inspire me confidence (no shame on the maintainers) and it relied to much on monkeypatching for my taste.
Implementing it myself was actually not hard and it helped me a lot to learn how OTEL works. And it also gave me confidence to instrument other part of the stack instead of relying on another dependency.
Reading this blog post I get the impression that OTel is going GREAT. So many languages with stable support, with equally stable development and progress across the board. For years my experience with otel was bad until a colleague spent a couple evenings learning more. Since then it has been wildly effective and useful. I struggle to read this post and agree with its own conclusion!
I only tried the TS one (there are two version iirc) and both of them were a mess to work with. Yes, it worked. No, it didn't help.
The app became much slower and hogged memory like it was free.
For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?"
Why are you doing this to people? You’re much better off with your own wrapper lib for features you use.
I’m fairly sure otel exists in order to prevent better things appearing: both by forcing new entrants to be compatible with the awful behemoth and sucking attention away from any attempt to make a better model. This benefits the incumbents.
I'm not sure if it's a datadog on windows servers issue but my only experience with that platform was that it was anything but turn-key. It was more broken by default, undebuggable and undocumented than anything else.