fix: allow tabs in the thematic break that ends a list item - #4087

Marked version: 18.0.11 (master, 4417582)

Markdown flavor: CommonMark

Description

A thematic break whose characters are separated or followed by tabs does not end an open list item.

Input (tabs shown as <TAB>):

- foo
_<TAB>_<TAB>_

Expected:

<ul>
<li>foo</li>
</ul>
<hr>

Actual:

<ul>
<li>foo<hr>
</li>
</ul>

A trailing tab is enough to trigger it for every break character, and with - the result is worse, because the line is then read as a setext underline:

- foo
---<TAB>

Expected <ul><li>foo</li></ul><hr>, actual <ul><li><h2>foo</h2></li></ul>.

The same input with spaces instead of tabs (_ _ _, --- ) is already handled correctly, and *<TAB>*<TAB>* happens to work only because nextBulletRegex catches the leading * first.

Spec

CommonMark 0.31.2, section 4.1 Thematic breaks:

A line consisting of optionally up to three spaces of indentation, followed by a sequence of three or more matching -, _, or * characters, each followed optionally by any number of spaces or tabs, forms a thematic break.

A thematic break is not paragraph continuation text, so it cannot be a lazy continuation line of the list item (section 5.2, rule 5) and closes the list. Spec example 57 (- foo / *** / - bar) already covers the space-only form, which marked passes. commonmark 0.31.2, the reference implementation already in devDependencies, produces the expected output above for all three tab variants.

Root cause

other.hrRegex in src/rules.ts, used by the list tokenizer to decide that a following line closes the item, allowed only spaces after each break character, while the block hr rule it is supposed to mirror already allows [ \t]. The break line therefore fell through to the paragraph-continuation path and stayed inside the item.

Spec suite

Before: 1801 tests, 1801 pass, 0 fail.
After: 1803 tests, 1803 pass, 0 fail (the two added tests). The CommonMark and GFM completion tables are unchanged, including the known failures (Entity and numeric character references 15/17, Links 84/90, GFM Autolinks 15/19 and 11/14, Disallowed Raw HTML 0/1). npm run test:unit (191 pass), npm run test:lint and tsc are clean.

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.