zmx: session persistence for terminal processes
78 points by erock
78 points by erock
Greetings! After a couple of months of R&D I finally reached a place with this project where I'm using it as a fulltime replacement for what I would normally use tmux for: session persistence of terminal processes
This essentially extracts the attach/detach functionality from tmux and turns it into its own tool. Instead of using tmux for windows, tabs, and splits, you would instead leverage your own window manager to handle that.
Another neat aspect of this tool is terminal state and history restoration using libghostty-vt. We use libghostty-vt to restore the previous state of the terminal when a client re-attaches to a session.
How it works:
- user creates session zmx attach term
- user interacts with terminal stdin
- stdin gets sent to pty via daemon
- daemon sends pty output to client and ghostty-vt
- ghostty-vt holds terminal state and scrollback
- user disconnects
- user re-attaches to session
- ghostty-vt sends terminal snapshot to client stdout
In this way, ghostty-vt doesn't sit in the middle of an active terminal session, it simply receives all the same data the client receives so it can re-hydrate clients that connect to the session. This enables users to pick up where they left off as if they didn't disconnect from the terminal session at all.
I've seen this trend pick up to avoid tmux/screen like tools. It's an interesting/new way of working from how I have been. I use mosh and of course then use tmux on the far side, so I get multiple windows.
I don't think there are terminal features I need that I would use, so the switch currently isn't worth the hassle to me, but this tool seems like it's finally possible to use this workflow.
I came from really terrible links, so mosh was like a miracle when I first started using it. Now my connections aren't usually speed limited, so I don't need mosh anymore, but habit you know. :)
Perhaps some terminal somewhere will get some new whizbang feature that I really want and make the switch, but until that time I don't think it's worth the hassle, but I'm glad tools like this exist to explore the problem space.
Or maybe I'll move to Arcan instead.
can you explain how it is different from shpool?
Great question!
shpool only supports a single client connected to a single session, zmx lets multiple clients connect to the same session.
zmx uses libghostty for better terminal state support (shpool has issues with terminal state restoration: https://github.com/shell-pool/shpool/issues/46)
shpool runs a single daemon for all terminal sessions, so if a single terminal session causes a crash all terminal sessions will crash. The same is true for any issues or load-bearing issues. zmx creates a daemon per terminal session so if a single session crashes or is under load it doesn't kill or bog the rest of your sessions.
shpool uses rust where zmx uses zig.
Thats about it! Philosophically they are highly aligned and like I mentioned in the README, I was heavily inspired by shpool
thanks - i do use ghostty and i remember needing to connect multiple clients for something, so i will check this out
The ssh and window explanations are really confusing. What exactly is the difference between this and just opening multiple terminal emulator windows and ssh I to a server and pick up processes backgrounded with for example nohuop?
Personally I have used screen for 20+ years and always have served me well. But I am curious about this as I agree with that point about redundant functionality.
What exactly is the difference between this and just opening multiple terminal emulator windows and ssh I to a server and pick up processes backgrounded with for example nohuop?
So I've never used nohup cmd & before but I did do research on how to detach/reattach using commonly installed tools -- including nohup. My take after reading posts about it was it's kind of hard / not designed for re-attach which is a critical piece of zmx. I'm curious, how would you re-attach to a nohup process?
I'm curious -- with this setup, how does one handle restoring multiple windows/panes/tabs? That is for me the killer feature of tmux (and wezterm): being able to save/restore groups of terminal windows.
I'm reminded of dtach, https://dtach.sourceforge.net/. There were a few others around that time that were lighter than GNU screen.
I'm kind of interested in how people are using their terminals. I come from the [mostly bad] old days where the first thing you did was compile and install some GNU tools including screen on vendor Unix (SunOS & Solaris but true for HPUX and Ultrix). tmux was an improvement over screen and I most didn't care what my local terminal emulator is capable of except for launching ssh to get into a persistent session on a work host. I appreciate 24-bit/truecolor and fast render but don't use terminal scrollback. I looked at Zellij but decided I didn't want to fight with its conflicts with emacs keybindings and try to adapt Zellij to match my muscle memory from tmux.
I use dtach for managing my shells on my remote development machine. I have some scripting set up so that basically each of my gnome-terminal tabs will connect to its own dtach session on the remote machine and it works pretty well. I like terminal scrollback and found that tmux and screen messed it up.
I think the main difference with zmx is that it doesn't just reconnect you to the same PTY, it tracks terminal state and replays that. I've been half writing something just like this recently, so I'm excited to take a look at this.
I tried zmx out and it seems to work well, congrats and thanks, I can see this being useful.
I did two "stress tests": first I created a session and splatted (an effectively unlimited) stream of random bytes, then detached, waited a bit, and attached from another terminal to see if it had "desynced" or broken the terminal in some way, and it did fine:
$ zmx a stress1
$ find / -type f -exec cat {} \;
<ctrl-\>
$ zmx a stress1
Then I created another session, output 100k of lines, each line an increasing number, detached, attached in a new terminal and scrolled back as far as I could. Then I subtracted the first line number I could see from the line number at the end of the output to get the amount of scrollback zmx seems to support:
$ zmx a stress2
$ for i in {0..100000}; do echo $i; done
...
100000
<ctrl-\>
$ zmx a stress2
<scroll back>
99359
I didn't check to see if the scrollback buffer is bytes or lines but got 641 lines in this case.
the interesting part for me is 'rehydrating' the terminal emulator. I'd love it if when I attached to zmx/tmux/whatever, the scrollback of the native terminal emulator were populated (and perhaps other kinds of state, too). When I tried zmx just now on wezterm, I could not scroll back, but that might be just my particular setup. What I'd like to see come out of this project is a protocol that all terminal emulators could implement to rehydrate their state from a remote source. Like how the kitty-keyboard protocol is used by some terminal emulators as a defacto standard.
Okay, I just pushed a fix, this was actually an oversight on my part: https://github.com/neurosnap/zmx/commit/53dacdc48a25e91807a19f5cfbb8e1107cb43315
Thank you for giving me a name for "alternate buffer", a concept I've been vaguely aware of but never had a name for.
Scrollback is one of the reasons why I wrote my own similar program too, but for me, I did an approach fairly different: when the alt screen is active, scrollback commands are forwarded to the application. So shift+page up, for example, is just sent as input instead of trying to do the local scrollback. Thus, the nested session can process it to do its own thing.
But this only works because both the top-level emulator and the inner attach emulator are based on my own code. By controlling the entire stack, I can do things like that... a thin outer shell forward commands knowing the inner shell uses the same implementation and can respond correctly. Does the same thing for like copy/paste - a middle click on the outside is forwarded to the inside. It reads middle click and requests paste from the outside, which then requests it from the system gui, which passes it back down the chain. Not the most efficient thing ever, but it does a reasonable job keeping up the illusion of actually moving the session to a new computer.
(also i could do some command like "push scrollback history" but meh, i'd rather display it lazily on demand. I wrote it when I literally paid by the kilobyte for mobile data and spent a fair amount of time on laptops from cars so i wouldn't want to send down $5 of data i didn't need...)
I’m going to look into this. I don’t have scrollback history explicitly set in ghostty so it might be a quick fix. Will post back when I see what’s going on
This rocks and I honestly think this workflow is the future. There was really too much friction with tmux for me and I could never get into it beyond the occasional tmux -CC session. I would literally rather re-SSH to a box each time and ctrl+r for my last command. Also great to see a serious use of libghostty-vt. Great work! I'll try this out when I get the chance.
I faintly remember arguing against the "you might not need tmux" post when it was posted, so I'm wondering if I would get something out of this (although I'm of course open to try it if I discover something)
double parsing doesn't necessarily halve the throughput. what nonsense is that? kitty is written in python and it's not particularly fast in my tests, i don't know what kind of cooked benchmark they came up with to make it look faster than everything else. you can most definitely implement some light parsing that adds negligible overhead when compared to kitty
but regardless, this hardly matters because you're going to run vim in it and type at human speeds, for which really almost every terminal is fast enough
The Kitty benchmarks and their methodology are public: https://sw.kovidgoyal.net/kitty/performance/
I assume this is in reference to the YT video (https://www.youtube.com/watch?v=yOK4EJDyjcM) in my blog post (https://bower.sh/you-might-not-need-tmux): I think that's totally fair. When I used tmux the latency issues I noticed were not tmux but the remote ssh connection itself.
Do you feel compelled at all by the other arguments for not using tmux? Namely, in order to use a terminal feature you need support for it in both your terminal emulator and tmux for it to work?
kitty is written in python...
Just to correct this:
~T/kitty[master]‡$ tokei -scode
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
Go 291 67367 61376 1207 4784
Python 231 71439 61337 1548 8554
C 134 67406 56778 3735 6893
C Header 96 40392 29604 8458 2330
kitty is primarily C, with an increasing amount of Go over time.