Why Markdown emphasis fails in CJK: A deep dive into CommonMark's delimiter rules

Thank you 👍

In my experience these are almost never used on the Web. The only place I've seen them are exam papers.

Aha thanks. There is a css feature for this too, like text-emphasis: dot filled #ff00ff;

I don't think anyone does this in the formats that allow it

Data point of one, but I do it. For instance <i lang="fr" title="French Onion Soup">Soupe à L’Oignon gratinee</i> (edited to add: or did I misunderstand your statement as meaning markup languages other than HTML?).

I would expect there are CSS selectors for language?

Kind of. You can select tags with the lang attribute (p[lang]), an exact language (p[lang="en-GB"]) or a more fuzzy matching (p[lang|="en"] for all dialects of English).

djot seems to handle both of the following cases correctly, if I'm not mistaken:

*마크다운(Markdown)*은

このような*[状況](https://example.com)は*

This is a good example of a markup format having been designed with not all human languages in mind, and making some languages harder to mark up as a result.

Some additional details I found by experimenting with the commonmark.js dingus:

• Though the article only mentions ** syntax, the same problem happens with the alternative __ syntax for strong emphasis, as well as with the * or _ delimiters for (normal) emphasis.

• There are workarounds for the problem.

  • In environments where embedded HTML is allowed (Markdown’s original use case), the formatting can be achieved with <strong> tags.
  • In environments where embedded HTML is filtered, you can add a visually-unnoticeable U+200A HAIR SPACE between the unrecognized ** delimiter and the adjacent ordinary character.
    • I tried adding a U+200B ZERO WIDTH SPACE, but that did not affect the parsing.

See those results by rendering this Markdown with the dingus:

Strong emphasis not parsed:

- **마크다운(Markdown)**은
- **마크다운(Markdown)**​은 (zero width space)
- __마크다운(Markdown)__은 (`__` delimiter)
- *마크다운(Markdown)*은 (`*` delimiter)
- _마크다운(Markdown)_은 (`_` delimiter)
- このような**[状況](...)は**
- このような**[状況](...)​は**
- このような__[状況](...)は__

Emphasis not parsed either:

- このような*[状況](...)​は*
- このような_[状況](...)は_

Strong emphasis parsed:

- <strong>마크다운(Markdown)</strong>은 (`<strong>` tags)
- **마크다운(Markdown)** 은 (hair space)
- このような<strong>[状況](...)は</strong>
- このような **[状況](...)は**

This type of misbegotten cleverness is unreasonably popular in lightweight markup languages (LMLs). It affects scriptio continua (where you’re not using a word separator) the most, but also other places: it’s perfectly reasonable to want to emphasise part of a word.

Markdown is an inconsistent mess, though at least these days you’re almost always dealing with the basic CommonMark rules, limiting the variation of the madness so that the biggest weirdness remaining is the difference between underscore and asterisk (a_b_c → a_b_c, a*b*c → abc). Some desired stylings are inexpressible, which is bad. Falling back to HTML is not always permitted, and even when it is it’s ugly.

reStructuredText has the same basic problem, but with more consistent rules and a universally-applicable workaround: backslash-space acts as a word boundary, so you can make a word **part**\ ly bold. It’s a weird syntax choice, though, because it’s not an escaped space like any other backslash escape—it doesn’t emit anything.

AsciiDoc defaults to constraining to word boundaries, but lets you double the delimiter to get unconstrained. This is… a choice. AsciiDoc has a lot of esoteric and idiosyncratic syntax. It’s terse and hard to learn, the Perl of LMLs.

I’ve been making and using my own lightweight markup language for five years or so, and I decided very early on on a philosophy of simple rules, even if nuanced rules may often be more convenient.

In my language, nothing cares about word boundaries, and *…* is just emphasis wherever it occurs. An unclosed delimiter will produce something like an HTML parse error (which don’t stop processing by default and have defined behaviour). a * b = c could be expressed in XML as something like a <em> b = c</em><?parse-error "unterminated *"?>.

reStructuredText frames the rationale for these features thus: “The inline markup recognition rules were devised to allow 90% of non-markup uses of *, ` [aside: I don’t think it’s possible to produce <code>`</code> here on Lobsters, which is a related problem in Markdown], _, and | without escaping.” I just think that’s a mistake of a goal. A simple rule is easier to understand and apply. People make errors in both directions all the time, you can’t really stop that, so I reckon simplicity is better. Especially when it’s maximally-flexible, and lets you write less code and parse faster.

---

A related area: hard-wrapping. At present, every markup language that supports hard-wrapping gets it wrong for all forms of scriptio continua, and also for cases like em dashes at the start or end of lines—and that was the issue over which I decided to make my own LML. They all turn the line break into a space, which is not always appropriate. HTML/CSS leaves scope for a better implementation, but no user agent has done anything better yet.

The cleverness angle also bites LMLs on paragraph boundaries, because of hard-wrapping. Markdown and reStructuredText both handle list markers in crazy but different ways. Consider https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#enumerated-lists:~:text=enumerators.-,For,dude.,-Caution. Too clever. This is also an area where I believe my LML is better than all before (in a more interesting way which I won’t describe here).