LogProc - A New Logs Datastore
4 points by Pentlander
4 points by Pentlander
I am always impressed at the solutions in this problem space (observability). I would be interested to see how this compares with all the other open-source solutions.
I do wonder, however, how much of these copiuous logs (750 TiB in this case) are actually useful. Are we solving the right problem?
The design is most comparable to Loki, I took a lot of inspiration from its design but thought the implementation could be simpler. It has fewer components than Loki so it’s easier to manage and scale. Unlike Loki, it has bloom filter and fst indexes to make UUID and full text queries usable. Then there are a bunch of low level impl details that make it faster, like storing the data as flexbuffers instead of raw strings.
To your second point, very very little of it, think of it as an insurance policy. But also that size is likely the uncompressed size and zstd compression was around 10x and there are ways to make it even smaller.
I built the prototype of this and led the design when I still worked there, I’m pretty happy with how it turned out.