Why .tar.gz files can't be combined with cat
11 points by gavinmorrow
11 points by gavinmorrow
ISTM that the issue isn't that the author lacks understanding of tar and gzip, but that the lacks understanding of cat: it was intended for simple text files, and won't work properly if- as a simple example- the first file is terminated by ^Z and subsequent software honours it as an EOF.
This is not true, cat is binary safe and not limited to text files. It will copy the "^Z" (0x1a) just as it does as any other byte. There is no EOF character in Unix, ^Z is a CP/M-ism that DOS copied.
Whenever stuff like this is brought up, I'm reminded of how silly it is that we collect things with tar before compressing. The idea of separating file bundling and compression is fine, but tar is such a bad fit for modern usecases, no? Or at least non-streaming usecases. Whenever I write an NPZ file I find myself wondering whether I should start using foo.zip.lz or whatever instead. With the inner zip being uncompressed, of course.
I didn't know TAR did this before, and that got me into a rabbit hole exploring NAR. Archiving is simple enough that TAR would stay relevant. Compression is better angle for size decreases. Though NAR seems to be experimenting with compressed blocks of NAR for more parallelism.
By the way, what are the non-streaming use cases? Aren't archive formats like these usually made for transfer?
A non-streaming use case is that you sometimes just want one of the files from an archive.
Tar is short for tape archive. It is specifically intended for streaming because the intended use case is to take data and create a stream that can be written to the tape. Adding in compression is composable in exactly the same way.
As I recall, both zip and tar put their dictionaries at the end of the file, because that's when you have already written them (PDF does the same - write all of the objects, then write the dictionary at the end).
Compressing an uncompressed zip file has the same problems as compressing a tar archive: you need to decompress it entirely to be able to get any file out. This is why zip does per-file compression: the dictionary is uncompressed (but is tiny), each file can be pulled out one at a time and used. This is why zip works nicely as a read-only filesystem (each file can be decompressed on demand as it's used) but tar.gz does not (you must decompress the whole thing to read one file).
Modern compressors such as zstd have a dictionary mode, where you give it a load of data and it generates something roughly like a Huffman encoding of the common sequences across the entire data set, then you can individually compress files referencing that dictionary, so they're independent but the common parts are in the dictionary. I'd like to see a ZIP-like format that used this, with a shared dictionary but the ability to decode individual files (and, ideally, individual blocks of files, with something like a 2 MiB granularity) independently.
Let's not forget the ultimate Unix hack of recursively copying a directory by tarring to standard input, piping that to a subshell that does cd and then untarring the standard output...
Not encouraging it, but doesn't deflate have a flush mode which is basically "reset this stream"? Feels like a little bitstream twiddling at the start/end of each tarball could join the streams without recompressing
shar enters the chat... Although
I’m working on a project that generates multiple .tar.gz archives, and I need to combine them into one final file. I thought I could just cat the bytes together, but that doesn’t work.
You could just tar the compressed files into one tape archive and later select out of it whichever .tar.gz file you need for further processing. However, doing that would not have taken you into this journey.
I recently stumbled on chapter-tgz, which makes specially crafted tar files that let you skip over “chapters” without having to gunzip the inner blocks (and lets you decompress in parallel) which is a neat trick.