Base84 deserves a place in file names
22 points by untitaker
22 points by untitaker
The accidental but very convenient avoidance of Windows special device names is very cute. However, the space savings are really tiny:
Base84 guarantees room for 197 bytes of input, compared with 191 for unpadded Base64.
Yeah, I don't really think it's worth the added implementation complexity to move from Base64 to Base84. One of the nice things about Base64 is that it's easy enough to roll your own implementation, so you can use it on any platform. Maybe there's a hyper specific reason why you need a handful of extra bits in a filename, but it must be a pretty niche circumstance where base64 wouldn't work.
Still, I appreciate that this implementation uses 5-character clusters so you can do encoding/decoding with a sensible algorithm. I've had the gross misfortune to interact with Base53 before, which cannot be encoded/decoded with chunking, you have to work with the entire value in memory at once.
Every character is acceptable in a filename on the usual Linux, macOS and Windows filesystems.
Except, Windows and macOS have case insensitive file systems so you risk clashes.
<pedantic> MacOS gives you the option but it's not the best idea as a lot of Mac software assumes insensitive filenames </pedantic>
Windows also gives you the option on a per-directory basis. Although, yes, some software won't be able to handle it. Especially software with many decades of history. DOS used to uppercase filenames and extensions and that convention seems to have carried over into early Windows, continuing long past the time it was actually needed. Whereas the modern convention is to use lowercase or else leave case alone. So modern software with some ancient internals can be internally inconsistent (which matters little if the filesystem is insensitive).
I keep source code on a separate volume with case sensitivity enabled, which lets me keep the root volume using the defaults.
The problem with these weird bases is the conversion method. "It emits groups of five characters." That's a non-obvious, arbitrary, and certainly not generic, constraint.
A specific conversion method must also be specified. For example, RFC 4648 base64 provides a conversion method. It has to because base64 doesn't fit exactly into the byte paradigm while still being a power of 2. Objective outsiders ("aliens from a distant planet) not using the byte paradigm will not come to the same conclusions as humanity. Base64 itself is inefficient because it has the constraint of base256: 7 bit ASCII encapsulated in the byte paradigm, base256. However, base64 is more obvious and less arbitrary than "Base84" since it is a power of 2. Base64 in itself is 100% efficient, as is any power of 2, assuming Shannon's informational foundation. Any non-power of 2 has waste, extra potential "unused codes". This is one of the foundational principals of Information Theory itself.
There is an alternative: arbitrary base conversion using the "iterative divide by radix" algorithm (see https://convert.zamicol.com to play with it). Arbitrary base conversion is generic. It only requires one variable: your chose output alphabet (given a known input).
I really really like the idea of BASE84, but may I propose excluding one more character? The comma ,? If you include this for BASE83, it can be used to encode strings in CSV files, and make parsing them pain free.
I suppose an alternative here is to simply use one of the excluded values as field separators; for example the pipe | is also a natural separator given its use in markdown tables.