@claytonlin1110 looks like it needs a bit of a cleanup and is missing a few tests. After that we should be good.

Findings

No remaining Critical or High issues found on the current head of PR #4308.

Missing High-Value Tests

A few worthwhile tests are still implied by the new API surface, even though I would not block merge on them at high severity:

• An end-to-end JSON fixture test with real Formula elements through elements_from_json() -> elements_to_md(), not just synthetic Formula(...) units.
• A regression test that create_file_from_elements(..., formula_markdown_style=...) actually propagates the style option through markdown output.
• A full positional-signature regression for elements_to_md(elements, filename, exclude_binary_image_data, encoding, normalize_formula) to lock the new parameter order.

Verdict

Merge

One non-blocking note: the branch still contains outputmarkdowndiff.txt, which looks like a committed scratch artifact rather than product code. I’m not counting that as a critical/high code issue, but I would remove it before landing for branch hygiene.

@PastelStorm Updated. Please review

@claytonlin1110 sorry for the drip :|

Code Review: claytonlin1110:feat/formula-markdown-latex

This branch bundles four distinct features: (1) Formula element rendering as $$ display-math blocks in Markdown, (2) embedded CMap stream parsing for CIDFonts in pdfminer, (3) PDF rendering delegation to unstructured-inference, and (4) standardize_quotes refactor via str.translate. Below are findings of Critical and High severity only.

Finding 1 — HIGH: LaTeX command substitutions fuse with adjacent characters, producing invalid commands

File: unstructured/staging/base.py — _normalize_formula_for_markdown

The function performs bare str.replace() for Unicode→LaTeX mappings (e.g. "∈" → r"\in", "≤" → r"\leq"). When the source text has no whitespace around the symbol (common in OCR output), the LaTeX command name fuses with the adjacent character, creating an undefined command.

Why it's risky: LaTeX/KaTeX/MathJax tokenize \ + consecutive ASCII letters as a single command name. \inS is parsed as the undefined command \inS, not \in followed by S.

Failure scenario:

Input:  Formula("x∈S")
After:  "x\inS"  →  KaTeX error: \inS is not recognized

Same for a≤b → a\leqb, a≠b → a\neqb, etc. This affects every non-whitespace-delimited math symbol — a very common case in OCR-extracted formulas.

Fix: Append {} after each LaTeX command to terminate the name: "∈" → r"\in{}".

Finding 2 — HIGH: Normalization runs before the auto-detection heuristic, changing wrapping decisions

File: unstructured/staging/base.py — _emit_formula_markdown

Normalization (_normalize_formula_for_markdown) runs before _formula_auto_use_display_math. The heuristic scores Unicode symbols ([∈∉≤≥≠≈×÷∞∑∫√∂∇]) at +2 each, but after normalization those codepoints are gone and replaced by \leq etc., which trigger the different \\[a-zA-Z]+ check (flat +3). This shifts scores at boundary conditions.

Failure scenario:

Text (85 chars, contains "where"):
  "E ≤ threshold where E is the energy and threshold was determined experimentally"

Raw score: ≤ gives +2, total=2. Prose threshold=3. Score 2<3 → plain text. ✓
After normalization: \leq triggers \\[a-zA-Z]+ → +3, total=3. Score 3≥3 → wrapped in $$. ✗

User sees a prose sentence incorrectly wrapped as display math.

Fix: Run auto-detection on the raw text, then normalize only the text that will be wrapped.

Finding 3 — HIGH: Normalization alters text even when formula_markdown_style="plain"

File: unstructured/staging/base.py — _emit_formula_markdown

When style == "plain", the function returns text — but normalization has already mutated it. A caller using plain style reasonably expects unmodified text.

if normalize_formula:
    text = _normalize_formula_for_markdown(text)  # already mutated
# ...
if style == FORMULA_MARKDOWN_PLAIN:
    return text  # returns normalized, not raw

Failure scenario: element_to_md(Formula("a − b"), formula_markdown_style="plain") returns "a - b" (Unicode minus silently replaced with ASCII hyphen) instead of the original "a − b".

Fix: Check style before normalizing, or skip normalization when style == "plain".

Finding 4 — HIGH: Bare except Exception swallows critical errors silently

File: unstructured/partition/pdf_image/pdfminer_utils.py — CustomPDFCIDFont.get_cmap_from_spec

except Exception:
    logger.debug("Failed to parse embedded CMap stream %r", cmap_name)

This catches MemoryError, RecursionError, and any bug in the new parsing code, logging only at debug level without a traceback. In production (log level INFO+), this is completely invisible.

Failure scenario: A CMap contains odd-length hex like <F>. bytes.fromhex("F") raises ValueError. The exception is swallowed, the font falls back to an empty CMap, and all text for that font is silently lost — with zero production log visibility.

Fix: Narrow to specific exceptions (ValueError, KeyError, zlib.error). At minimum add exc_info=True and log at warning.

Finding 5 — HIGH: Partial CMap returned when mapping budget is exceeded

File: unstructured/partition/pdf_image/pdfminer_utils.py — _parse_embedded_cmap_stream

When a cidrange exceeds the remaining mapping budget, the code uses continue (skip this range, keep processing subsequent ranges). This can return a CMap with arbitrary gaps — some code points decode correctly, others are unmapped.

Failure scenario: A CJK CMap has ranges R1 (5K entries), R2 (130K entries), R3 (1K entries). R1 is accepted, R2 is skipped (exceeds 131K cap), R3 is accepted. The CMap is returned with R1+R3 but missing R2. Since code2cid is truthy, it's trusted. Most CJK characters (in R2) decode to wrong CIDs → garbled text output.

Fix: Return an empty CMap() once the budget is exceeded, rather than a partial map with holes. A partial CMap is worse than no CMap because the caller trusts non-empty code2cid.

Finding 6 — HIGH: exactly_one(filename, file) validation dropped

File: unstructured/partition/pdf_image/pdf_image_utils.py — convert_pdf_to_image

The old _render_pdf_pages called exactly_one(filename=filename, file=file) to reject calls where both arguments are provided. The replacement render_pdf_to_image from unstructured-inference only checks if both are None — it does not reject both-truthy.

Failure scenario: A caller passes both filename="old.pdf" and file=<in-memory bytes of new.pdf>. The old code raises immediately. The new code silently uses filename (via filename or file), ignoring the file bytes entirely. The caller gets page images from the wrong PDF with no error.

Fix: Add exactly_one(filename=filename, file=file) inside convert_pdf_to_image before delegating.

Finding 7 — HIGH: CustomPDFResourceManager.get_font duplicates parent cache/dispatch logic

File: unstructured/partition/pdf_image/pdfminer_utils.py

The override re-implements the cache lookup, strict-mode check, and subtype extraction from PDFResourceManager.get_font. If pdfminer's internal get_font logic changes (e.g., adding pre-processing, new font subtypes, or changed caching), the override diverges silently.

Risk: A pdfminer version bump adds a pre-processing step (e.g., resolving indirect references) that the override bypasses for CIDFont types. CIDFont construction receives un-resolved spec entries.

Missing High-Value Test Cases

1. LaTeX command boundary: Test Formula("x∈S") — expected $$\nx \in{} S\n$$ or similar, currently produces broken \inS.
2. Normalization vs. heuristic ordering: Test a prose-length formula with exactly one Unicode math symbol + a prose keyword, verifying it's NOT wrapped.
3. CMap budget exhaustion with multiple ranges: Test that a CMap exceeding the mapping cap returns an empty CMap (not a partial one).
4. Both filename and file provided to convert_pdf_to_image: Should raise ValueError.
5. Encrypted CMap stream: Test that decipher path in _decode_pdfstream_with_limit handles missing objid/genno.

Positive Notes (not in scope, but worth acknowledging)

• The standardize_quotes refactor is strictly correct — fixes two silent data-loss bugs (U+201C, U+2018 were dropped by duplicate dict keys) and the ",," entry was dead code in the old implementation. No issues found.
• The thread-safety concern about removing _pdfium_lock is not a regression — render_pdf_to_image in unstructured-inference has its own lock, and consolidating to one lock is actually an improvement.
• The embedded CMap parsing logic is well-bounded with size caps and mapping limits (modulo Finding 5 above).
• The test coverage for the new CMap parsing is thorough.

Verdict: Not Merge

Findings 1–3 are correctness bugs in the core Formula→Markdown feature (the primary purpose of this PR). The LaTeX command boundary issue (Finding 1) will produce broken output for common OCR inputs. The normalization-before-heuristic bug (Finding 2) causes incorrect wrapping decisions at boundary conditions. These should be fixed and re-verified before merging. The remaining findings (4–7) in the pdfminer and pdf_image_utils areas add production risk but are individually addressable.