feat(engine): join Arabic through its presentation forms - #538
Conversation
|
A/B result, as promised in the description.
Every average sits inside the harness's declared noise band (~5–10% on a laptop), and the two runs disagree on sign for three scenarios with no code difference on the LTR path between them — the first run's |
|
Pushed An author can now say where Arabic must not join. U+200C forbids a join the letters The shaper needed almost nothing for the non-joiner: it is an opaque non-joining They get their own predicate rather than joining the bidirectional controls: these say Two other corrections in the same commit:
Tests: four on the shaper (non-joiner breaks a join, joiner forces one, neither survives Full reactor gate green. |
Arabic letters change shape by position, and a font does that through OpenType GSUB — which a PDF content stream never executes: showText walks the cmap and nothing else. The only way to reach the joined forms there is to hand the font their code points directly, from the presentation-forms block the bundled Arabic family was chosen for carrying. The shaper is a pure text-to-text mapping: a per-letter table of the four positional forms, the four lam-alef ligatures, and the joining walk that picks a form from what precedes and follows. Vowel points and the bidirectional format characters are transparent — they sit between letters without breaking the join, exactly as Unicode's joining rules say, and a direction mark that DID break a join would visibly tear the word apart. It is idempotent, since the forms lie outside the base block, and text with no Arabic letter is returned as the same instance after one early-exiting scan — the path every existing document takes. The inverse mapping is deliberate API: a font that lacks a form can draw the base letter — unjoined but readable — instead of losing the text to '?'. That degradation lands with the glyph seam, in the next change; nothing consumes the shaper yet.
Shaping runs before measurement — on the logical lines of the plain and markdown paths, at token construction on the inline path — because the contextual forms have their own advance widths and the engine's contract is that what is measured is what is drawn. Text without an Arabic letter passes through as the same instances. Each backend again does only what its medium requires. The PDF draws the forms, since its content stream will never run the font's own shaping. PowerPoint gets the base letters back at its seam: it shapes Arabic itself, and frozen presentation forms would end up in a file users search and copy from. Word was never handed forms at all. Mirroring lives with reversal at the PDF seam and nowhere else, for the same reason reversal does — both PowerPoint and Word apply UAX #9 L4 themselves, and a mirror baked into the span would come out double-mirrored there. A font that carries the Arabic letters but not the forms — the GSUB-only families — now degrades at the glyph seam to unjoined base letters instead of '?', which costs the joining rather than the text; a lam-alef ligature decomposes back into its two letters. The proofs read the content stream: every Arabic glyph on the page is a presentation form and none a base letter, a parenthesis before Hebrew is drawn facing what it encloses, and the slide text carries base letters with no form leaked into it.
The recipe's "not done yet" section becomes a section on how joining works — through the presentation forms the font carries, with the degradation to unjoined base letters named for fonts that lack them. The example's Arabic row caption follows, and its committed preview is regenerated so the published render shows the joined forms it now produces.
…alef Two ordering bugs, both found by review before anything shipped. The glyph seam may degrade a lam-alef ligature into its two letters, and it appends them in logical order — into a string the handler had already reversed for display, which landed the pair swapped on the page. The handler now sanitizes the logical text first and reverses last, so every substitution the seam makes keeps its letter order. The degradation also warns once per font now: losing the joining silently left an author no clue why their Arabic came out unjoined. The ligature is formed only for an adjacent lam-alef pair. A vowel point between the letters belongs to the lam; folding the pair into one glyph stranded the mark after the ligature, and the inverse mapping the slide backend relies on put it back on the wrong letter — the slide's text then differed from what the author wrote. A vocalized lam-alef renders as two joined letters instead: unligated, and exactly restorable, which the round-trip test pins. Alef maksura's deliberate divergence from ArabicShaping.txt gets the comment that stops the next reader from fixing it, the recipe names the covered block (the Persian and Urdu extensions render unjoined for now), and the fully-qualified names become imports.
…iews Without the attribute, core.autocrlf treats a committed preview PDF as text on checkout and rewrites its line bytes in the working tree. The file then shows as permanently modified, which corrupts the published preview and blocks every branch switch across a change to it — the A/B benchmark harness was the first to hit the wall.
…seline Every other test in this feature asserts one mechanism — span order, run direction, the code points in the content stream — and each can pass while the page still looks wrong, because none of them looks at the page. This one does. The document is the acceptance case from the issue: Hebrew with an embedded Latin word and a year, Arabic joined with a mirrored parenthesis pair around Latin, and a paragraph left to AUTO. The baseline was blessed after reading the render, and any visible change to how these scripts lay out fails it, whichever mechanism caused it. The pixel budget matches the sibling regression test's, which was settled against observed cross-platform antialiasing drift; a real regression here repaints whole lines and clears it by orders of magnitude.
The zero-width non-joiner and joiner are how Unicode lets an author override contextual joining — U+200C forbids a join the letters would otherwise make, U+200D forces one — and both were deleted before the shaper, the only thing that reads them, ever ran. They are category C like any other control, so the sanitizer that runs first removed them along with the rest, and a document could not express the distinction at all. The shaper needed almost nothing for the non-joiner: it is an opaque non-joining character, so the lookups that walk letters already came to the right answer once it survived. The joiner is the one that had to be taught, since it stands in for a letter that is not there. Both are consumed during shaping, because past that point they are code points no font can encode — measured as one thing and drawn as a substitution mark. Text with no Arabic in it never reaches the shaper, so the glyph seam drops them there instead. The joining controls get their own predicate rather than joining the bidirectional ones: these say nothing about direction, only about whether two letters connect. Also: - Auto-size measured a highlight run unshaped while the token it builds from that run is shaped, so a chip was sized against a string the layout never uses. Both paths shape now. This is not currently visible end to end, because auto-size does not shrink a paragraph holding a chip at all for any script, which is its own defect. - The degraded-shaping warning is keyed on the font rather than on (font, code point). A font that carries no presentation forms carries none of them, so the second letter said nothing the first had not, and a document turned one fact about the font into a warning per letter. - The changelog claimed mirroring per UAX #9 L4 without the scope the implementation has always documented: the punctuation that occurs in documents, not the whole mirroring table.
355a469 to
6c1efb1
Compare
Stacked on #536 (→ #534 → #535) — review and merge in that order. Closes the Arabic-joining item of #140.
Why
Arabic letters change shape by position, and a font does that through OpenType
GSUB— which a PDF content stream never executes:showTextwalks the font'scmapand nothing else. With #536 the letters already stood in the right order; they stood unjoined, every one in its isolated form. The presentation forms Amiri was selected for carrying (#535) are the one way to reach the joined shapes in a PDF, and this is the shaping step that maps to them.What changed
ArabicShaperis a pure text-to-text mapping: a per-letter table of the four positional forms for the base block (U+0621–U+064A + tatweel), the four lam-alef ligatures, and the joining walk. Vowel points and the bidirectional format characters are transparent to the join, as Unicode's joining rules say. Shaping is idempotent, and text with no Arabic letter passes through as the same instance.It runs before measurement — on the logical lines of the plain and markdown paths, at token construction on the inline path — because the forms have their own advance widths and the engine's contract is that what is measured is what is drawn.
Each backend keeps doing only what its medium requires:
toBaseLettersat its seamMirroring (UAX #9 L4:
()[]{}<>«»‹›) lives at the PDF seam next to the reversal and nowhere else — both office engines apply L4 themselves, and a mirror baked into the span would come out doubled there.A font that carries the letters but not the forms — the
GSUB-only families, Scheherazade New among them — now degrades at the glyph seam to unjoined base letters instead of?, with one warning per font: the joining is lost, the text is not. A lam-alef ligature decomposes back into its two letters. The handler sanitizes before reversing so that decomposition keeps its letter order on the page.Two deliberate trades, both tested: a vocalized lam-alef (a vowel point between the letters) renders as two joined letters rather than the ligature, because folding the pair would strand the mark and make the slide text differ from what the author wrote — the round-trip is exact instead. And a character-split of an over-long Arabic word keeps mid-word forms at the seam.
Also:
*.pdf binaryin.gitattributes— without it, checkout treats a committed README preview as text and rewrites it in the working tree, corrupting the preview and blocking branch switches across a change to it.Verification
Full reactor gate green; no snapshot or visual baseline moved.
The proofs read the PDF content stream: every Arabic glyph on the page is a presentation form and none a base letter; a parenthesis before Hebrew is drawn facing what it encloses; the slide text carries base letters with no form leaked into it (
PptxRtlParagraphTest); degradation is exercised against a mockedGSUB-only font, including the two-letter ligature decomposition. Shaper table tests pin every letter's forms, the four ligature variants, transparency of harakat and direction marks, idempotence, and the same-instance fast path. ~30 new tests across core, render-pdf and render-pptx.The published
text-directionexample preview is regenerated — its Arabic row now shows the joined forms it produces.Performance: the left-to-right cost is one early-exiting scan per line/token. A first A/B run against the base branch showed sign-flipping deltas with one scenario above the harness's noise band; a clean re-run is in flight and its table will be posted as a comment before this merges.
Known limits