How JPEG Screenshots Defeated Our Beautiful H.264 WebCodecs Pipeline
16 points by susam
16 points by susam
Ubuntu packages are missing random features. Always check. Or just build from source like it’s 2005.
One of the things that frustrates me conceptually about a lot of package managers is that the delta from "I want to use the package as-is" to "I want to make one small flag change" is massive. Obviously you'll have to build from source, but it's annoying you can't just apt install grim --flag jpeg or whatever.
Gentoo's USE flag system is basically exactly this in concept.
I've never used Gentoo but it doesn't surprise me that they have a better way to handle this considering you're building from source anyway.
The syntax can be somewhat obtuse, but nix makes this, if not easy, at least possible:)
Yeah, I've used it in my personal config for similar things, it's pretty nice (even if I have to remember how to do it every time).
Although, to be fair, avoiding the subsequent rebuild of dependent packages is sometimes possible but always only via things considered dark arts.
(But I do like that even binary-based use of Nix is basically «let's plan how to compile all that… oh wait there is a cache that has this exact glibc build already done»)
40 Mbps to stream a mostly static desktop screen is insanely high. Even 10 Mbps is too high for such usage. As they can accommodate a 1s delay, they should also be able to use variable bitrate. They should start with 1 Mbps and check the result.
The issue is that text (likely the overwhelming majority of what's important to them) ends up blurry and hard to read when going down to lower bitrates of H.264. Newer codecs like AV1 might help, but that's still far from perfect. Proper damage tracking could probably help as well, but at that point you're reinventing VNC.
Having fallback so robust it'd almost work in Netscape is nice, but the H.264 stream should be salvageable.
You have to have backpressure when flooding WebSocket with data, and unfortunately the bufferbloated Internet won't do it for you. But you don't need to reinvent TCP congestion control. Just make the client send back the timestamp or index of the frame/segment it just received. When the server sees the client lags behind, it can lower the bitrate, skip unsent segments, or restart the stream. You could also sync the clock between server and client, and have the client restart connection when it sees it's downloading stale frames.
We almost shipped a hilarious bug.
When you stop sending video frames, the WebSocket becomes basically empty. Just tiny input events and occasional pings.
The latency drops dramatically.
Our adaptive mode sees low latency and thinks: “Oh nice! Connection recovered! Let’s switch back to video!”
Video resumes. 40Mbps floods the connection. Latency spikes. Mode switches to screenshots.
Latency drops. Mode switches to video.
Latency spikes. Mode switches to screenshots.
This is where it becomes real interesting imo.