Why I prefer human-readable file formats
21 points by runxiyu
21 points by runxiyu
The points presented in the article are very valid, but there’s a counter-point: how would you choose to represent something like an image file in a human-readable manner? Sure you could encode it into a JSON array, but that would be very inefficient.
The apparent simplicity of human-readable formats can also be deceiving. Underspecified formats like JSON are notorious for having differing behaviour across parsers. On the other hand reading a binary format engineered for simplicity can be done in a remarkably compact way in code, see rxi’s simple serialisation system—and by “compact” I mean machine-sympathetic.
I personally think mechanical sympathy is just as important for longevity as is maintainability by humans.
There’s more counter-points than that e.g. text-based formats are a mire of parsing ambiguities and thus parsing differentials (which is very much a point-against in terms of auditability, and manual correction for that matter as seemingly irrelvant and possibly invisible characters become technical lynchpins), the “simple tooling” is usually an alias for broken, and the purported efficiency is nonexistent. Compression is hardly an argument as it trades efficiencies, and can be applied to binary formats besides.
The version control friendliness is also… dubious. I have seen changes performed by automated tooling to text-based format, and the diff may as well not exist.
the “simple tooling” is usually an alias for broken
The formats that are self-describing and readable get reused enough to get reusable tooling, which eventually gets good. Unique binary formats are much more likely to have incomplete and partially broken tooling.
how would you choose to represent something like an image file in a human-readable manner?
I have used the plain-text PBM/PGM/PPM image formats (examples) to create and modify example inputs during the development of image manipulation code. (Not to counter anything you’re saying, as this is a niche use case, but it might be interesting to some.)
# ####### ## #
# # # # # #
# # # #
# # # ### ## ###
# # # # # # # #
# # # # ##
# # # # ##
# # # # # #
### ### ## ### ### #
(Although my agreement isn’t human readable because it’s not accessible. Maybe I should have stuck with the command line that made it: pbmtext -space 1 '^ This.' | pnmcrop -plain | tail -n +3 | tr 01 \ #)
i like to extend “readable” to mean “legible in common tools”
one of those is a text editor. others are image editors, hex editors, etc.
there’s a chicken and egg issue, though: even utf8 text files weren’t human readable until text editors started widely supporting them (unless they were in the ascii subset)
i think sqlite probably counts as readable, if you don’t jam anything too nasty into the values. zip/tar file full of any other readable format is readable.
What file formats does this disqualify? If I can open an Excel sheet in LibreOffice, or inspect an elf file with objdump, are they readable?
it’s a spectrum. excel is more readable than a custom binary format, but less readable than CSV (which excel can open). if you don’t care about the proprietary part, excel is both readable and writable in many cases. it’s a common format when you need people to edit data, and don’t want to build a UI
elf is less readable. even when you pull sections out, the content is binary with no hint on how to interpret it. with a zip file, you get file extensions to help with that
very unreadable would be: custom binary formats that don’t align well for hex editors, don’t self-describe, and use ad-hoc compression
i’d also argue that encoding a normal map as JSONL or something would be unreadable. the format itself is readable, but the content is more intelligible when you can visualize it
Images almost always come with metadata, so if it’s a json with all of them and other attributes and then one attribute for the pixel-data itself, then that’s maybe a mix of text and binary, but still. Not saying it’s a good idea.
Underspecified formats like JSON are notorious for having differing behaviour across parsers.
Let’s not pretend like there are no different parsers behaviors when it comes to images. PNG bombs were a thing as well, just as one example. :)
I find the thing gltf (a fairly recently designed format for models due 3D graphics) does somewhat cute. It’s a JSON document immediately followed by a binary blob. The metadata is small and is put in the json object. The mesh and textures being big are in the binary blob and the json document describes where to find them relative to the start of the binary blob.
There are a bunch of features in gltf that probably make it very confusable though. The main one I know of is that data that is normally stored in the binary part can actually be put directly in the json (encoded as base64 or something I think).
Underspecified formats like JSON are notorious for having differing behaviour across parsers.
Supporting arbitrary key-value pairs and being popular enough tends to lead to implentation differentials for repeated keys for logical reasons, no matter the encoding.
mechanical sympathy
One of the points in the article is that the meaning of mechanical sympathy sometimes changes across use cases and hardware generations.
image file in a human-readable manner would be very inefficient
For 256-colour palettes XPM and PPM do not even have much overhead compared to any other uncompressed format…
but there’s a counter-point: how would you choose to represent something like an image file in a human-readable manner?
There are human-readable image formats, off the top of my head I remember XPM. It was used by SimCity for example. They are from an era before GPU. ej. https://github.com/SimHacker/micropolis/blob/master/MicropolisCore/images/airport.xpm. I’m sure there were others with different trade-offs. XPM is meant to be easily included in your C/C++ application.
The apparent simplicity of human-readable formats can also be deceiving. Underspecified formats like JSON are notorious for having differing behaviour across parsers.
That is a property of being underspecificed, not human readable.
I think a large part of it is, human-readable formats are great for documents you want to edit.
Binary formats aren’t really editable without specialized tools. They are very suitable for generated files (think protobuf), and for communicating with other machines.
If the main purpose of the data is to communicate from machine A to machine B, then a human-readable format might not be the right choice: we just get an innefficient, ambiguous communication.
And so we combat this how? By minifying it, and losing all the benefits we had. Minified JS and HTML might be “readable” but you’ll be hard-pressed to make heads or tails out of them.
Meanwhile a configuration file is meant for communicating with a program sure, but largely it’s for you and other people using the tool. You could write a GUI to edit a binary configuration file, but the efficiency of parsing that may very well not be worth it.
I think SMTP and HTTP are two protocols that are largely text based and would nowadays benefit from being more machine readable
This longevity isn’t just about the distant future, it’s about practical resilience over years and decades. When software vendors pivot, get acquired, or simply discontinue products, human-readable formats remain accessible. Your data doesn’t become hostage to a company’s business decisions or technical roadmap. Text-based formats have survived multiple generations of computing paradigms because they’re built on fundamental, stable foundations.
Ironically, I’m currently dealing with a case where human-readable formats are worse at this: some debug tracing data at $work is stored as JSON, and that means that changing the name of a field causes the data to become unparseable. If we stored it in protobuf this wouldn’t be an issue.