wl-kbptr: Control the mouse pointer with the keyboard on Wayland
22 points by lenny
22 points by lenny
How waylandesque: it works in everything but Gnome and KDE.
I definitely share you frustration.
If there’s one thing that was lost going from Xorg to Wayland, it’s being able to write a tool or an application and expect it to just work everywhere. Beyond the core Wayland protocols, the compositors can implement whatever they want.
For wl-kbptr to work I need to be able to show on top of everything. That’s done with the Layer Shell protocol. KWin implements it but for Mutter they decided they didn’t want it.
Then I need to be able to move the cursor and be able to take screenshots (for the detection feature). Once again, compositors do this thing very differently. On most Wlroots compositors, they implement the WLR Virtual Pointer and WLR Screencopy interfaces. Neither KWin nor Mutter implement or have plans to implement these. Instead, they use their own APIs through D-Bus — e.g. org.kde.KWin.ScreenShot2 to do take screenshots with KWin.
One thing I’ve considered was to write a KWin script, have it find the current window, start the program which would take the screenshot and then return the position for the script to move the pointer. The problem here is that unless I’m mistaken it can’t move the pointer directly from these scripts (you can get the position though…). One thing I could use (or copy from) is ydotool, but that’s emulating an input device which from my tests is not quite reliable when the idea is to move the pointer to a specific location.
So now I guess another approach would be to go with XDG Desktop Portal and try their Remote Desktop and ScreenCast API. I didn’t fully explore this idea but after playing with a bit with it, there’s going to be limitations if it works (and some compositors don’t implement these). I’ve also tried the Screenshot API, but it’s too slow — it can only take the whole desktop and then you have to load it from a file which is pretty inefficient.
Sounds like you’ve explored the possibilities pretty thoroughly. Thanks for the tool - I would have found something like this useful when I had a small laptop that I used on a sofa! Trackpad quality is pretty variable…
KDE’s Kwin has their own version of this feature built-in, so the user should not need this tool for it.
I mean, as a desktop user I try to use the same software across windows / mac / linux whenever I can. Ideally I would be able to do that across different linux compositors instead of learning one software per compositor.
It’s totally fair to want this from a user perspective. From a developer perspective, having to reliably support a wide range of platforms (most of which you don’t use) is not fun at all. That’s the reason why I kept the scope small by only targeting Wayland and even then it’s still fragmented. Having said that, I’ve been looking at ways to make it reliably work on KWin but it’s been quite frustrating.
I wonder if you could just make it “redirect” things to KWin, so it would just use KWin implementation through your application.
Do you mean the thing in System Settings → Accessibility → Mouse Navigation which enables to move the mouse with the keypad? If not, I’m not sure which tool we are talking about (I looked in Plasma 6.3.5).
I asked around and there is no API for that, sadly. But there’s two things you might be able to do:
Hacky, but make your program show up as a device and the commands send keypad inputs. Requires the accessibility thing to be enabled in settings though.
You could try to use remote desktop portal API for moving mouse around.
For example in dbus, you can find service org.freedesktop.portal.Desktop and under that, there is org/freedesktop/portal/desktop/org.freedesktop.portal.RemoteDesktop with bunch of different methods.
I think you could start a session through that portal, then use NotifyPointerMotion etc to move mouse around. This actually might work on Mutter too, if they have the portal implemented.
edit: If you don’t feel like making that, I could give it a go this weekend perhaps :) (can’t promise anything but I’m intrigued by the problem)
Thanks! That seems to confirm the conclusion I end up with.
I tried the org.freedesktop.portal.RemoteDesktop API a few days ago and:
ScreenCast API. A workaround would be to move the cursor to the top left as much as possible (hoping the user doesn’t have any weird display positioning) and then we can move the cursor to the position we want.As for Mutter, unless we find a way to display on top of all the applications, there’s no much that can be done.
I’ve just created an issue (wl-kbptr#44) in the repository. If you want to have a go, have fun. I might try eventually but probably not this week.
It would be really nice if tools which, by their own description, are centered around wl-roots protocols used the wlr- prefix instead of wl-.
I was a bit too ambitious initially when I decided on a name I guess. I’ve considered changing the description but then a lot of non-wlroots compositor implement the necessary protocols. That’s why I settled on having a compatibility section in the README.
At the end of the day, if you have a Wayland tool that doesn’t just use basic windows, it will comes with a lot of asterisks anyway. If you take a look at the Awesome Wayland list, they will state the required protocols next to the tools’ description which I think it the right approach.
On the flip side, I haven’t given up on supporting other compositors (mainly KWin), it’s just not a priority.
Thanks for sharing my project! It’s nice seeing people having an interest in my work.
It’s been interesting to see how niche projects like this one can spread as initially I just had this repository on GitHub and I didn’t talk about it. People still started using it and it even got packaged for some distributions.
I was looking for something exactly like this ever since my touchpad broke. The touchscreen doesn’t always work properly for some apps, and I’m not about to carry around another mouse.
Not having to carry around a mouse is precisely why I made this tool in the first place.
I started this project as a way to be able to use my computer from my sofa when watching things on the TV. I had a wireless keyboard and didn’t want to have to deal with a mouse (or buy one of these keyboards with a trackpad that end up never being used). Whilst I could have stopped at using simple bindings with Sway, I thought it would be interesting to see how Wayland apps work.
This is pretty neat. I’m used to moving the mouse with keys defined at the keyboard firmware level, but I’m also used to Vimium-type hints in the browser, and this seems more like the latter.
The compositor support picture looks to me like “supports wlroots and Smithay”. Is that more or less accurate?
Smithay seems to have the required wlr protocol implemented, but I tried to use it on a compositor (COSMIC) written in Smithay and it failed to detect the protocol.
That’s the funny thing I’ve observed with Wayland compositors: having the underlying library support the protocol is not enough. The compositor itself needs to add the support even if the library will provide what’s necessary to do so. Of course, someone could come up with a library that works differently and be more hands-on so that wouldn’t be required.
The compositor support picture looks to me like “supports wlroots and Smithay”. Is that more or less accurate?
I’ve gave up on giving general rule like that since at the end of the day and as things stand, the individual compositors need to implement the protocols themselves. Even if they do, their implementation might be broken — that was initially the case with Hyprland but now it works perfectly.
A good indication is looking at the compatibility section in the README and see if your compositor supports the required protocols. If someone finds a compositor with which it works that is not on that list, I welcome PRs.