Apache Flink's 95% Problem
4 points by topicpartition
4 points by topicpartition
Upvoted for the "you probably don't need Flink" premise, cancelled my vote because the proposed solution is ClickHouse which is also a 80%-problem technology: it's also a 18-wheeler truck that requires a ton of tuning and operational toil.
Ah, it's written by a managed Clickhouse provider, figures.
yeah it is biased, but I read it as 65%+ don't need clickhouse or flink hence why I shared it
in my experience, you either get biased vendors pushing for flink or others pushing against; I don't really see regular devs blog about Flink a lot at all
In our team, click house is way more expensive than Kafka + flink, didn't dig deep into this though.
out of curiosity, how do you track the spend? Is it a vendor, or do you have metrics for your self-hosted solution that lets you calculate what % of your AWS bill is from Clickhouse?
Our conclusion was that Flink is a huge knowledge cliff for a relatively small use case. For our team size it's not worth maintaining the skills and experience in another highly complex technology.
what do you use instead?
We have one managed Flink instance running but it doesn’t scale organisationally.
For the use case that we have where we need to assemble denormalized objects together from a CDC stream to send off to the search index, right now our go to solution is to write a small web app that assembles by querying from the source of truth database. It’s less efficient and also less declarative but it seems like a solution that more people in the organisation can understand.
Maybe we’ll have a better solution in the future but it seems like the cost/complexity trade-offs are not in our favour.
Interesting that even with a managed instance you say it's a huge knowledge cliff
The cliff is mostly getting Typescript/Python engineers to be able to deal with writing and testing a Java application.
The managed instance is quite limited but running our own would require doubling down on a huge amount of additional knowledge.