Stop MITM on the first SSH connection, on any VPS or cloud provider
25 points by JoachimSchipper
25 points by JoachimSchipper
This is cool because it can be automated!
As a manual way I've been using the cloud provider's console to confirm the server's SSH fingerprint out of band.
I don't manage that many cloud instances, so having a couple manual steps for provisioning is fine
Yes, that works. I was just using a provider that didn't wire that up, and also developing a script to automate setup...
An alternative approach if you have automated your DNS zone:
This flow allows the server to keep its private SSH host key without rotating it.
Most DNS providers doesn't allow this kind of granular one time access tokens, but you could have a simple internal web service which validates tokens and then issues API calls on your behalf with a permanent non-scoped token, which the SSH server doesn't have access to.
That's creative! I agree that that would work; a one-time token to a custom service is quite flexible. I'd be inclined to generate an SSH certificate instead of writing to a DNSSEC'ed domain, but at that point you're discussing which solution better fits your particular context.
Are you aware of any software / providers which allow you to generate tokens that are that flexible, or would that need a bit of custom development?
I hadn't thought about SSH certificates – that's a nice addition when looking at the available options :)
No DNS providers comes to mind, but there might be some out there with that kind of scoped tokens (but I haven't checked). As a software developer, anything below 50 lines of code is faster than researching different vendors and migrating. I admit that my "sounds easy and fun" might be perceived differently by others.
As a (former) sysadmin, that sounds like "one more service to manage". But I agree that writing the code would be fun! And you do eventually want to automate updating DNS, although updating DNS from the administrator workstation is easier to reason about w.r.t. security.
Aside: ssh-init-vm is basically three ssh invocations. And I did have fun writing it ;-).
This is pretty neat!
(By the way, the date on the article has the month and day muddled up.)
Thanks! It's always a pleasure to work with OpenSSH.
I've fixed the month/day; thanks again!
Thank you for sharing!
I'm glad you mentioned the metadata service at 169.254.169.254, which seems more-or-less consistent across VM providers (which we can see from the cloud-init source by looking at the various cloudinit/source/DataSource*.py entries.)
Personally, I am getting a bit tired of cloud-init as a result of its design as well as its limitations. Additionally, I am interested in unifying system setup, across locally hosted virtual machines (using QEMU,) remote machines, containers, and physical hardware.
The arch-boxes project shows us how an ArchLinux cloud-init image is created, and it's a very simple collection of shell scripts. If we adapt the methodology used in arch-boxes, either using guestfish or a µvm, we can create a shell script that can be used for provisioning an OCI-compliant image, a disk image for use in QEMU or a cloud provider, or used to provision a new physical machine. (As in, the exact same script!)
With this, and a few QEMU flags, we can replicate your approach without the dependency on cloud-init. As far as I know, we cannot use systemd.system-credentials to pass the transient host keys—there's only credentials for ~/.ssh/authorized_keys (e.g., ssh.authorized_keys.root)
However, we can create a unit file that runs either during the initrd (if we are using systemd for this) or along with the systemd-firstboot.service. Said unit file can be built into the image itself or injected transiently using the systemd.extra-unit.* credential and enabled using the systemd.wants=… kernel command-line option. For QEMU, we could simulate the presence of the metadata service with -netdev user,id=metadata,net=169.254.0.0/16,dhcpstart=169.254.0.15,guestfwd=tcp:169.254.169.254:80-cmd:… entry (though we will likely need to enable the interface that is created, which may best be done with some transient unit file we have created.)
This gives us a lot of flexibility (at relatively low complexity) for performing consistent system setup across various types of “machines.”
In practice, for this particular task, I think probably the best approach would be to have your image creation tool (your arch-boxes equivalent) create your machine image with fixed host keys and install your custom host key rotation script as a SystemD service—triggered on first reboot/shutdown?
Note that for ArchLinux, if you enable the systemd HOOK in /etc/mkinitcpio.conf, you can/have to write SystemD unit files for tasks you want to perform in your initrd.
In practice, I have found this is only very slightly clumsier than writing {/etc,/usr/lib}/initcpio/hooks.
However, enabling systemd-networking and systemd-resolved in your initrd is quite easy, meaning your initrd can be tasks with system start-up responsibilities, and you can schedule them before you pivot to the root file system!
Obviously this probably won't work for physical hardware like a laptop, which probably needs NetworkManager or similar to connect to wifi, but it works quite well for QEMU VMs and hosted VMs, and it seems like a lot of system start-up responsibilities fit very nicely into this space.
The goal of all of this is, of course:
cloud-init