Optimizing memory usage in a markdown parser

I wouldn’t think a Markdown parser needed memory optimization, since the largest md files it’s likely to encounter are a few megabytes, and the parse tree's only going to be a few times that since there’s way more text than formatting instructions.

The first thing I would work on is streaming rather than building a full tree (SAX vs DOM.) Memory overhead is very small that way. An intermediate is to build a tree only for the current top-level block element — Markdown doesn’t have much high level nesting, so a document is just a sequence of blocks. The exceptions are lists and tables, but those typically don’t get too huge.

Nonetheless these are good optimizations in general. I’m surprised they used pointer compression but stuck with 8-byte offsets; you save a lot going down to 4 (as V8 does) and 4GB isn’t much of a limitation when the files you’re parsing are a thousandth that size.