On Perfetto, Open Source, and Company Priorities

I'll talk to to my colleagues about it so I can try to have a more complete view, and I'll try to write a bug report next week on that ~

Please do! Will allow us to track these things a lot better.

It would be nice to "follow" a process by clicking on it to see where it ends up being scheduled. IIRC, that's not available (or I missed it :D)

You can definitely do this, Perfetto wouldn't be very useful for Android engineers if this wasn't possible :) There's actually many ways you can get to this:

1. from the CPU view, if you find one instance of the thread (you can search for that instance), you can hover over it and see all the other places in the thread view
2. In the process view, you can see for a thread, the scheduling state over time (running, runnable etc) and if you click on any of them, the bottom panel will show you the CPU where it's scheduled.

Some description of fields within the SQL in the documentation. IIRC, I wanted to output the time in "waiting for schedule" and "schedule" and i ended up trying to decipher the fields by hand and trying them out using SELECT * and such.

Would very much suggest using the Standard Library docs (https://perfetto.dev/docs/analysis/stdlib-docs). Specifically, exactly what you are looking for can be found in the sched.time_in_state module (https://perfetto.dev/docs/analysis/stdlib-docs#sched-time_in_state):

INCLUDE PERFETTO MODULE sched.time_in_state;

SELECT state, time_in_state
FROM sched_time_in_state_for_thread
JOIN thread USING (utid)
WHERE thread.name = '<my thread here>'

Of course the question would be "how would I even know to get to this section of the stdlib docs" and that's a very fair point! I ties with your question below around case studies.

(IIRC) the config assumed some things like some already activated compile time options for Linux kernel which are probably present on Android but not necessarily on normal linux servers (IIRC, it was PSI or something related to frequency scaling, but it's been some time).

Yeah this is a fair request. I agree that this would be nice but even we (Perfetto team) are not experts in this. Someone would need to sit down and reverse engineer ftrace event -> config option and then figure out whether ot not Linux distributions set this/not. It's a lot of thankless work though I agree it would be very useful to have this!

I find the docs nice for tutorials for initial usage and more in depth things, but I'd like something akin to (in the context of a linux server or maybe workstation for instance) :

Yes this is something we definitely want to invest more time into. We recently wrote https://perfetto.dev/docs/case-studies/scheduling-blockages which is in the context of Android but 99% of it can apply to desktop Linux. There's also https://perfetto.dev/docs/getting-started/android-trace-analysis which is less applicable to Linux but still might have some useful snippets.

We fully want to do many more things like this going forward too!

More for my personal use-cases (that I haven't got the time to get to yet), I'd like to use it to get more insight into early boot performance issues (more than just systemd-analyze blame/critical-chain).

Early boot on Linux becomes a very tricky problem because of, as you pointed out before, the number of different options/flags which could be set both at the kernel and the userspace level which change what sort of things you need to enable/collect for this. This might be a trickier one for us to help with without spending a bunch of time looking into early boot on Linux ourselves :)

Thanks for the pointer!

I think one thing that really helped jj is that their culture and team was set up from day 1 to be not just "source control for inside Google" but also "an alternative/complement to git for open source": that is, reading through their history, it was a personal project which initially was worked on by one person, gained traction and then a team was formed around it. It's a lot easier to set things up in the right way at the start than retroactively change the structure later on.

That said, there's 100% things we can take inspiration from. Something I will look more into and try to learn from :)

Oh interesting! I never heard of it before. I'll keep it in mind whenever I get a perf issue to debug :)