On Perfetto, Open Source, and Company Priorities
31 points by rayhaan
31 points by rayhaan
Perfetto is such a useful tool. I certainly wish it was more usable on usual linuxes.
Author here! As I said in the bottom of the post, I would really love to understand where the gaps in usability are on general Linux distributions. Not guaranteeing we'll be able to prioritize it but it does let us understand where we stand.
It also means if we do get to dedicate some personal time, we can jump into building the things people care about the most!
I'll talk to to my colleagues about it so I can try to have a more complete view, and I'll try to write a bug report next week on that ~
To sum up a bit what we did though : we're a midsize company and we wanted to try out sched_ext.
And perfetto is (AFAIK) one of only tool to debug that kind of thing.
I also had another use-case recently when investigating performance issue for wireguard-go. It uses many goroutines (and so, threads) and scheduling ends up being a bottleneck.
Note that I'm not a Perf engineer so, there probably are some parts where it shows I don't have the good "reflexes" of one.
From my testing (it's been a couple of months so, take it with a grain of salt) :
SELECT * and such.More for my personal use-cases (that I haven't got the time to get to yet), I'd like to use it to get more insight into early boot performance issues (more than just systemd-analyze blame/critical-chain). Also, I'm interested in using to improve my laptop battery life (so i can also extend its lifetime). And that one is probably very similar conceptually that what you'd do on an android since battery devices.
Basically (for my personal use-cases), improving baseline performance and extending my devices battery/lifetime by improving their resource usage (you know, the utopian idea of reducing someone's impact on the environment).
I'll talk to to my colleagues about it so I can try to have a more complete view, and I'll try to write a bug report next week on that ~
Please do! Will allow us to track these things a lot better.
It would be nice to "follow" a process by clicking on it to see where it ends up being scheduled. IIRC, that's not available (or I missed it :D)
You can definitely do this, Perfetto wouldn't be very useful for Android engineers if this wasn't possible :) There's actually many ways you can get to this:
Some description of fields within the SQL in the documentation. IIRC, I wanted to output the time in "waiting for schedule" and "schedule" and i ended up trying to decipher the fields by hand and trying them out using SELECT * and such.
Would very much suggest using the Standard Library docs (https://perfetto.dev/docs/analysis/stdlib-docs). Specifically, exactly what you are looking for can be found in the sched.time_in_state module (https://perfetto.dev/docs/analysis/stdlib-docs#sched-time_in_state):
INCLUDE PERFETTO MODULE sched.time_in_state;
SELECT state, time_in_state
FROM sched_time_in_state_for_thread
JOIN thread USING (utid)
WHERE thread.name = '<my thread here>'
Of course the question would be "how would I even know to get to this section of the stdlib docs" and that's a very fair point! I ties with your question below around case studies.
(IIRC) the config assumed some things like some already activated compile time options for Linux kernel which are probably present on Android but not necessarily on normal linux servers (IIRC, it was PSI or something related to frequency scaling, but it's been some time).
Yeah this is a fair request. I agree that this would be nice but even we (Perfetto team) are not experts in this. Someone would need to sit down and reverse engineer ftrace event -> config option and then figure out whether ot not Linux distributions set this/not. It's a lot of thankless work though I agree it would be very useful to have this!
I find the docs nice for tutorials for initial usage and more in depth things, but I'd like something akin to (in the context of a linux server or maybe workstation for instance) :
Yes this is something we definitely want to invest more time into. We recently wrote https://perfetto.dev/docs/case-studies/scheduling-blockages which is in the context of Android but 99% of it can apply to desktop Linux. There's also https://perfetto.dev/docs/getting-started/android-trace-analysis which is less applicable to Linux but still might have some useful snippets.
We fully want to do many more things like this going forward too!
More for my personal use-cases (that I haven't got the time to get to yet), I'd like to use it to get more insight into early boot performance issues (more than just systemd-analyze blame/critical-chain).
Early boot on Linux becomes a very tricky problem because of, as you pointed out before, the number of different options/flags which could be set both at the kernel and the userspace level which change what sort of things you need to enable/collect for this. This might be a trickier one for us to help with without spending a bunch of time looking into early boot on Linux ourselves :)
Look into how JJ is run! It's a Google project, funded by internal source control, and yet manages to not be hostile to external users and contributors, unlike pretty much every other Google OSS project I've seen :P
Thanks for the pointer!
I think one thing that really helped jj is that their culture and team was set up from day 1 to be not just "source control for inside Google" but also "an alternative/complement to git for open source": that is, reading through their history, it was a personal project which initially was worked on by one person, gained traction and then a team was formed around it. It's a lot easier to set things up in the right way at the start than retroactively change the structure later on.
That said, there's 100% things we can take inspiration from. Something I will look more into and try to learn from :)
Have you seen systing? It's an interesting BPF based tracer that outputs to Perfetto.
Oh interesting! I never heard of it before. I'll keep it in mind whenever I get a perf issue to debug :)
Oh yeah I forgot about this irritation ages ago. I assumed that I just hadn't searched hard enough to find the way this is done.
Thank goodness it's finally unlocked -- great job, Perfetto team!