How do you manage SSH keys?
31 points by mt
31 points by mt
Hi! Im interested in practices around key management.
Do you just use one key for everything, or a single key per host, or even one per use case?
Do you store them ~/.ssh, or in 1password or similar? Do you use ssh-agent or prefer specifying -i each time?
Et cetera. Please share your practices, especially if you feel that they are “unorthodox” but useful to you.
I either use Secretive (Mac only) to store my private key in the Secure Enclave and therefore have a key per device, or I use Tailscale SSH and don’t have to worry about SSH keys at all
+1 to this. I only use Secretive now, but I don't need to distribute my keys. I use a Yubikey as a CA and sign each device's public key: https://jamesog.net/2023/03/03/yubikey-as-an-ssh-certificate-authority/
I had no idea this was possible! This looks great, my 8 or so SSH keys are getting unwieldy.
macos supports this natively these days.
https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb4acf
+1 to secretive on my macbook.
Otherwise, I use bitwarden now that it has first-class support for SSH keys. their ssh agent is acceptable enough for making dotfiles easily portable to a new desktop-ish device.
I'm a bit lazy and don't really do "one key per device" and moreso one key per secrets medium. I played with yubikey ssh key/auth for a bit, but it's just a bit too clunky.
Just learned about Secretive, thanks. The 1Password ssh agent was the last thing keeping me there. I might try migrating away.
Wow, super interesting.
If you lose your machine you lose your key, I assume? How do you handle that?
By securely storing backup keys offsite?
By the way the SE protects from this kind of situation. So the HW doesn’t matter as much as the key itself. And the keys are accessible only by you.
Another negative is that you can’t work “remotely” by SSHing to that computer since the biometric 2FA will fail.
Post inspired by me looking in shame at my mess of a .ssh directory, in which lies a trove of keys, with no memory of what half of these are keys to, but too afraid to delete.
So my practices are not working that well, but in general i try to compartmentalize, using a key for one “section”, usually a host.
Key per host definitely needs a wrapper script on generation, no way anyone can keep track of that. There's no real security advantage unless they're all encrypted with different passwords though.
What threat model are you defending against?
If you use IdentitiesOnly and specify keys for domains/subnets in .ssh/config, you make cross-context identity matching not absolutely free for privacy-attacking adversary (especially between leaky contexts, like forges where some metadata API shows user keys). This forces listing the use cases in .ssh/config, making tracking feasible, too!
Otherwise… well, if things like Debian key generation story are expected to happen again but have shorter timespans, such key management limits the impact. Although, I guess, you still need to keep track of per-key targets, as knowing what is the affected keys but not knowing how to rotate them sounds more pain than gain.
Separate hardware tokens (Yubikeys) for personal and work. Keys never live on any machine and are portable. Never going back.
I'm planning to do the same. The latest release of libssh (0.12) added support for using FIDO/U2F keys, so once I've added support for that to the terminal I use I'll try switching. Will have to get a second Yubikey as a backup sometime, but I've been planning to do that anyway to backup the non-SSH things I already use my Yubikey for.
How do you manage backups?
Old job had us generate them on "never networked" machines and then imported to the key.
I do the same thing. That way I have 2 hardware tokens (main and backup) with the same key and then a separate offline backup.
I have some opinions.
I wrote ssh-tpm-agent to store hardware bound SSH keys. https://github.com/Foxboron/ssh-tpm-agent. Blog post: https://linderud.dev/blog/store-ssh-keys-inside-the-tpm-ssh-tpm-agent/
Next up is that I don't want long-lived ssh keys. I want short-lived ssh certificates bound to the device. I wrote ssh-tpm-ca-authority to try and solve this, but I was not super happy with how it turned out. https://github.com/Foxboron/ssh-tpm-ca-authority.
These days I run a smallstep CA in my home infrastructure, but I want this to issue short-lived hardware bound ssh certificates.
My current goal is to try and repurpose the device-atttest-01 ACME challenge to issue ssh certificates. For this to work I wrote my own attestation CA that slots into step-ca, and wrote up my own acme client and one agent that currently only does PKCS11 client mTLS certificates to my browser. All of this is very WIP and I plan to write up more things about this soon.
https://github.com/Foxboron/attezt
My goal is to hack support for a Content-Type: ssh-certificate header to the ACME certificate retrieval chain in smallstep to issue ssh certificates, and then ram these into my ssh-tpm-agent through the agent socket.
We'll see how far I get.
I have a 1Password-managed SSH key that I use for pretty much everything. For company-managed infrastructure, we use short term SSH key certificates issued by Vault for granting and managing access.
I used to recommend Teleport over Vault but since the AGPL switch it becomes harder in organizations that are AGPL-averse
The other side of that coin is that SSM actually works on non-AWS hosts (although at this moment it still uses AWS for the control plane) as well as allowing connectivity without necessitating inbound traffic to the hosts
I use gpg-agent and only have 2 yubikey-backed keys that I use (1 device/key is stored for backup in a fire safe). I followed the drduh YubiKey Guide which is excellent.
One key for work one key personal. On yubikeys. With the copy printed out on paper in a folder with all other recovery stuff.
I'm not a fan of the per-host or other multiple keys approach. If you're tempted, you should know exactly why you're going that way and be able to explain the threat model for it / what's improved. (My reason for splitting out work is that they may require rotating the key at some point for BS compliance reasons and I don't want to rotate my personal key)
I use ssh-agent with keys stored in KeepassXC. Works better than I'd thought it would.
I do that with a couple keys for different identities, and my ssh config determines which one is used based on host rules.
The key (heh) part is you can point IdentityFile to the public key so the private doesn't need to be on disk.
Yeah I have different personal and work keepasses, the idea being I can just hand over the database to the next guy when I leave the company.
A trick I learned for git is:
git clone -c core.sshCommand="ssh -i ~/.ssh/mykey_ed25519" [url]
This configures the cloned repo to use the specified key to talk to the remote. No need to mess with ssh-agent or .ssh/config. Particularly useful if you have keys for multiple Github accounts.
I use the 1Password SSH agent to log in to my servers (Windows, macOS, Linux). Works remarkably well and I don't have to store ~/.ssh keys locally.
IMO this is the best option if you want access to your keys in a distributed fashion (ie from multiple hardware devices). Carrying a Yubikey around sounds like a PITA to me.
I use tailscale ssh, I’m already authenticated by the VPN so it doesn’t make sense to re-authenticate again (and in the very few cases where it doesn’t, check mode forces a re-auth). For the few servers not on Tailscale, a SSH key derived from the authentication subkey of my PGP key.
By the way, you can use your own OpenID Connect provider for authenticating to Tailscale. I’m using Codeberg (with TOTP 2FA enabled); see Codeberg as an OIDC Provider for Tailscale.
A long, long time ago I wrote a post about this. It's mostly still relevant: https://nerderati.com/2011-03-17-simplify-your-life-with-an-ssh-config-file/
The tl;dr: you can use ~/.ssh/config for most basic use cases. These days password managers such as 1Password change the game a little bit, though, so you might want to investigate that.
I used this guide to help me set up SSH keys (and GPG) on Yubikeys. This approach also allows for backups (multiple keys). https://github.com/drduh/YubiKey-Guide
I have one key per «context», different keys on different client devices. Keys are backed up as «append-only data». They live in ~/.ssh, listed in ~/.ssh/config with appropriate targets, and IdentitiesOnly is on. I use ssh-agent for unlocking once per session (where hibernation counts as breaking the session). I do have some scripting integration for unlocking the password manager and loading keys into ssh-agent, but key file storage is independent.
I used to use one key, one machine.
these days I use my gpg key/agent, which is on a security key. I have my master key on one hardware token, and signing/auth/encryption keys on one I carry. Offline paper backups in safe.
this means if anything happens I can get a new yubikey/nitrokey/whatever, and authorize it from my offline master. it should work everywhere
I have 3 keys: personal, work, git-sign, they all live in ~/.ssh, backed up in my KeePassXC and on a separate flshdrive. In my ~/.ssh/config I set which keys(and not only) to use for each host.
I have one SSH key pair per device, stored only on said device. The passphrase is stored in Bitwarden and the OS' keychain so the SSH key is unlocked automatically upon login. Since I have multiple devices and add those to the relevant services (e.g. both my desktop and laptop have access to GitHub, my servers, etc) I don't worry about backing up the SSH keys, as in the worst case I can just generate a new one and bootstrap it using another device.
One per system, I sync public keys with scripts that pull from my public website, GitHub/gitlab post them publicly as well, when systems go away, I delete the public key from the site and they revoke when synced
Proton Pass, one master password I use nowhere else but have it memorized.
I wasn't aware they had this functionality, so if it interests others, too: https://protonpass.github.io/pass-cli/commands/ssh-agent/
although they sure do like doing this bullshit https://github.com/protonpass/pass-cli
I store them on my yubikeys, using gpg mode not the newer certificate-based mode for ssh. I have a distinct ssh key for each yubikey, which allows me to rotate them when the yubikeys are physically destroyed (my cats found one, a while back...). The private keys never exist on a computer, they are only ever held on the hardware tokens.
I use gpg-agent in its ssh-agent emulation mode, which allows me to forward the agent as usual.
For my own stuff, where I can configure sshd, I use epithet. Where I need some stable key, instead of being able to use certs, I keep my keys in a yadm secret in my dotfile repo.
I usually have files named according to the "username@host" scheme and an ssh_config that picks them up based on the scheme. If username@host doesnt exist, it falls back to some username@default key (first id_ed255yaddayadda, then id_rsa etc)
I have two keys in my KeePass database: One for git signing and git server authentication and one for connecting to my server via SSH.