Skip to content

feat(engine): join Arabic through its presentation forms - #538

Open
DemchaAV wants to merge 7 commits into
feat/rtl-bidi-reorderingfrom
feat/rtl-arabic-shaping
Open

feat(engine): join Arabic through its presentation forms#538
DemchaAV wants to merge 7 commits into
feat/rtl-bidi-reorderingfrom
feat/rtl-arabic-shaping

Conversation

@DemchaAV

Copy link
Copy Markdown
Owner

Stacked on #536 (→ #534#535) — review and merge in that order. Closes the Arabic-joining item of #140.

Why

Arabic letters change shape by position, and a font does that through OpenType GSUB — which a PDF content stream never executes: showText walks the font's cmap and nothing else. With #536 the letters already stood in the right order; they stood unjoined, every one in its isolated form. The presentation forms Amiri was selected for carrying (#535) are the one way to reach the joined shapes in a PDF, and this is the shaping step that maps to them.

What changed

ArabicShaper is a pure text-to-text mapping: a per-letter table of the four positional forms for the base block (U+0621–U+064A + tatweel), the four lam-alef ligatures, and the joining walk. Vowel points and the bidirectional format characters are transparent to the join, as Unicode's joining rules say. Shaping is idempotent, and text with no Arabic letter passes through as the same instance.

It runs before measurement — on the logical lines of the plain and markdown paths, at token construction on the inline path — because the forms have their own advance widths and the engine's contract is that what is measured is what is drawn.

Each backend keeps doing only what its medium requires:

Backend Receives Why
PDF the forms, then mirroring + reversal at the draw seam its content stream runs no shaping and applies no bidi rule
PPTX base letters, restored via toBaseLetters at its seam PowerPoint shapes Arabic itself; frozen forms would sit in a file users search and copy from
DOCX untouched logical text Word was never handed forms at all

Mirroring (UAX #9 L4: ()[]{}<>«»‹›) lives at the PDF seam next to the reversal and nowhere else — both office engines apply L4 themselves, and a mirror baked into the span would come out doubled there.

A font that carries the letters but not the forms — the GSUB-only families, Scheherazade New among them — now degrades at the glyph seam to unjoined base letters instead of ?, with one warning per font: the joining is lost, the text is not. A lam-alef ligature decomposes back into its two letters. The handler sanitizes before reversing so that decomposition keeps its letter order on the page.

Two deliberate trades, both tested: a vocalized lam-alef (a vowel point between the letters) renders as two joined letters rather than the ligature, because folding the pair would strand the mark and make the slide text differ from what the author wrote — the round-trip is exact instead. And a character-split of an over-long Arabic word keeps mid-word forms at the seam.

Also: *.pdf binary in .gitattributes — without it, checkout treats a committed README preview as text and rewrites it in the working tree, corrupting the preview and blocking branch switches across a change to it.

Verification

Full reactor gate green; no snapshot or visual baseline moved.

The proofs read the PDF content stream: every Arabic glyph on the page is a presentation form and none a base letter; a parenthesis before Hebrew is drawn facing what it encloses; the slide text carries base letters with no form leaked into it (PptxRtlParagraphTest); degradation is exercised against a mocked GSUB-only font, including the two-letter ligature decomposition. Shaper table tests pin every letter's forms, the four ligature variants, transparency of harakat and direction marks, idempotence, and the same-instance fast path. ~30 new tests across core, render-pdf and render-pptx.

The published text-direction example preview is regenerated — its Arabic row now shows the joined forms it produces.

Performance: the left-to-right cost is one early-exiting scan per line/token. A first A/B run against the base branch showed sign-flipping deltas with one scenario above the harness's noise band; a clean re-run is in flight and its table will be posted as a comment before this merges.

Known limits

  • Shaping covers the base Arabic block; the Persian and Urdu extensions (پ چ ژ گ …) render unjoined and break joins with covered neighbours. Named in the recipe.
  • A mid-word style boundary (bold half-word) shapes each fragment separately, producing word-edge forms at the seam.
  • Header/footer, watermark and debug-label text is not shaped.

for (int index = 0; index < logicalLines.size(); index++) {
String line = logicalLines.get(index);
String shapedLine = ArabicShaper.shape(line);
if (shapedLine != line && shaped == null) {
@DemchaAV

Copy link
Copy Markdown
Owner Author

A/B result, as promised in the description. ./scripts/ab-bench.sh -a feat/rtl-bidi-reordering -b feat/rtl-arabic-shaping -r 3 -p smoke — three interleaved runs per branch, medians compared, all-LTR corpus so this measures what existing documents pay for the shaping scan.

Scenario Avg Docs/s First run (pre-fix commit) Avg
cv-template +2.32% −2.09% −1.06%
engine-simple −5.58% +5.70% +7.87%
feature-rich +5.11% −4.85% +0.06%
invoice-template +2.44% −2.36% +10.20%
long-token +0.55% −0.59% −6.73%
proposal-template +2.84% −2.69% +5.93%
vector-rich +0.26% −0.19% +4.24%

Every average sits inside the harness's declared noise band (~5–10% on a laptop), and the two runs disagree on sign for three scenarios with no code difference on the LTR path between them — the first run's invoice +10.2% is +2.4% here. That is noise's portrait, not a regression's; the real regression this feature line once had (+10…+17% consistently across every scenario and stage at once) looked nothing like this. Left-to-right cost of shaping is one early-exiting scan per line/token, pinned by the same-instance test.

@DemchaAV

Copy link
Copy Markdown
Owner Author

Pushed 355a4695, which closes the joining-control gap.

An author can now say where Arabic must not join. U+200C forbids a join the letters
would otherwise make and U+200D forces one; both are category C, so control sanitizing
deleted them before the shaper — the only thing that reads them — ever ran, and a
document could not express the distinction at all.

The shaper needed almost nothing for the non-joiner: it is an opaque non-joining
character, so the lookups that walk letters already reached the right answer once it
survived the seam. The joiner had to be taught, since it stands in for a letter that is
not there. Both are consumed during shaping — past that point they are code points no
font can encode, measured as one thing and drawn as a substitution mark — and the glyph
seam drops them for text that never reaches the shaper at all.

They get their own predicate rather than joining the bidirectional controls: these say
nothing about direction, only whether two letters connect.

Two other corrections in the same commit:

  • Auto-size measured a highlight run unshaped while the token built from it is shaped, so
    a chip was sized against a string the layout never uses. Both paths shape now. I could
    not make a regression test for it go red, and the reason turned out to be a separate,
    larger defect: auto-size does not shrink a paragraph holding a chip at all, for any
    script. Measured with the Latin word Internationalization at page widths 90–260
    (10pt margins) — plain text goes 7.4 → 11.6 as the column widens while the chip stays
    at 221.99 / 184.99 / 147.99, constant multi-line blocks. That is pre-existing and
    unrelated to Arabic; it is tracked separately rather than folded in here.
  • The degraded-shaping warning is keyed on the font rather than on (font, code point).
    This PR's description promised one warning per font; the code gave one per distinct
    letter, so a document turned a single fact about the font into dozens of identical
    warnings.

Tests: four on the shaper (non-joiner breaks a join, joiner forces one, neither survives
into shaped text, neither lets lam-alef ligate), two on the sanitizer for both seams, and
one end-to-end in RtlGlyphOrderTest that renders beh + ZWNJ + heh and compares the
drawn glyphs against the joined form — the piece no unit test of either half could show,
since each was correct on its own.

Full reactor gate green.

Arabic letters change shape by position, and a font does that through
OpenType GSUB — which a PDF content stream never executes: showText walks the
cmap and nothing else. The only way to reach the joined forms there is to
hand the font their code points directly, from the presentation-forms block
the bundled Arabic family was chosen for carrying.

The shaper is a pure text-to-text mapping: a per-letter table of the four
positional forms, the four lam-alef ligatures, and the joining walk that
picks a form from what precedes and follows. Vowel points and the
bidirectional format characters are transparent — they sit between letters
without breaking the join, exactly as Unicode's joining rules say, and a
direction mark that DID break a join would visibly tear the word apart.

It is idempotent, since the forms lie outside the base block, and text with
no Arabic letter is returned as the same instance after one early-exiting
scan — the path every existing document takes.

The inverse mapping is deliberate API: a font that lacks a form can draw the
base letter — unjoined but readable — instead of losing the text to '?'.
That degradation lands with the glyph seam, in the next change; nothing
consumes the shaper yet.
Shaping runs before measurement — on the logical lines of the plain and
markdown paths, at token construction on the inline path — because the
contextual forms have their own advance widths and the engine's contract is
that what is measured is what is drawn. Text without an Arabic letter passes
through as the same instances.

Each backend again does only what its medium requires. The PDF draws the
forms, since its content stream will never run the font's own shaping.
PowerPoint gets the base letters back at its seam: it shapes Arabic itself,
and frozen presentation forms would end up in a file users search and copy
from. Word was never handed forms at all. Mirroring lives with reversal at
the PDF seam and nowhere else, for the same reason reversal does — both
PowerPoint and Word apply UAX #9 L4 themselves, and a mirror baked into the
span would come out double-mirrored there.

A font that carries the Arabic letters but not the forms — the GSUB-only
families — now degrades at the glyph seam to unjoined base letters instead
of '?', which costs the joining rather than the text; a lam-alef ligature
decomposes back into its two letters.

The proofs read the content stream: every Arabic glyph on the page is a
presentation form and none a base letter, a parenthesis before Hebrew is
drawn facing what it encloses, and the slide text carries base letters with
no form leaked into it.
The recipe's "not done yet" section becomes a section on how joining works —
through the presentation forms the font carries, with the degradation to
unjoined base letters named for fonts that lack them. The example's Arabic row
caption follows, and its committed preview is regenerated so the published
render shows the joined forms it now produces.
…alef

Two ordering bugs, both found by review before anything shipped.

The glyph seam may degrade a lam-alef ligature into its two letters, and it
appends them in logical order — into a string the handler had already
reversed for display, which landed the pair swapped on the page. The handler
now sanitizes the logical text first and reverses last, so every
substitution the seam makes keeps its letter order. The degradation also
warns once per font now: losing the joining silently left an author no clue
why their Arabic came out unjoined.

The ligature is formed only for an adjacent lam-alef pair. A vowel point
between the letters belongs to the lam; folding the pair into one glyph
stranded the mark after the ligature, and the inverse mapping the slide
backend relies on put it back on the wrong letter — the slide's text then
differed from what the author wrote. A vocalized lam-alef renders as two
joined letters instead: unligated, and exactly restorable, which the
round-trip test pins.

Alef maksura's deliberate divergence from ArabicShaping.txt gets the comment
that stops the next reader from fixing it, the recipe names the covered
block (the Persian and Urdu extensions render unjoined for now), and the
fully-qualified names become imports.
…iews

Without the attribute, core.autocrlf treats a committed preview PDF as text
on checkout and rewrites its line bytes in the working tree. The file then
shows as permanently modified, which corrupts the published preview and
blocks every branch switch across a change to it — the A/B benchmark harness
was the first to hit the wall.
…seline

Every other test in this feature asserts one mechanism — span order, run
direction, the code points in the content stream — and each can pass while
the page still looks wrong, because none of them looks at the page.

This one does. The document is the acceptance case from the issue: Hebrew
with an embedded Latin word and a year, Arabic joined with a mirrored
parenthesis pair around Latin, and a paragraph left to AUTO. The baseline
was blessed after reading the render, and any visible change to how these
scripts lay out fails it, whichever mechanism caused it.

The pixel budget matches the sibling regression test's, which was settled
against observed cross-platform antialiasing drift; a real regression here
repaints whole lines and clears it by orders of magnitude.
The zero-width non-joiner and joiner are how Unicode lets an author
override contextual joining — U+200C forbids a join the letters would
otherwise make, U+200D forces one — and both were deleted before the
shaper, the only thing that reads them, ever ran. They are category C
like any other control, so the sanitizer that runs first removed them
along with the rest, and a document could not express the distinction
at all.

The shaper needed almost nothing for the non-joiner: it is an opaque
non-joining character, so the lookups that walk letters already came to
the right answer once it survived. The joiner is the one that had to be
taught, since it stands in for a letter that is not there. Both are
consumed during shaping, because past that point they are code points no
font can encode — measured as one thing and drawn as a substitution
mark. Text with no Arabic in it never reaches the shaper, so the glyph
seam drops them there instead.

The joining controls get their own predicate rather than joining the
bidirectional ones: these say nothing about direction, only about
whether two letters connect.

Also:

- Auto-size measured a highlight run unshaped while the token it builds
  from that run is shaped, so a chip was sized against a string the
  layout never uses. Both paths shape now. This is not currently visible
  end to end, because auto-size does not shrink a paragraph holding a
  chip at all for any script, which is its own defect.
- The degraded-shaping warning is keyed on the font rather than on
  (font, code point). A font that carries no presentation forms carries
  none of them, so the second letter said nothing the first had not, and
  a document turned one fact about the font into a warning per letter.
- The changelog claimed mirroring per UAX #9 L4 without the scope the
  implementation has always documented: the punctuation that occurs in
  documents, not the whole mirroring table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants