regexlib ships as a single, dependency-free header. The Unicode tables it needs
are generated into regexlib.h from the Unicode Character Database (UCD) by
the scripts here — there is no runtime or build-time dependency on any other
library.
| Feature | Data | Source file |
|---|---|---|
\w \d \p{L|N|P|M} |
General_Category | UnicodeData.txt |
| grapheme clusters (UAX #29) | Grapheme_Cluster_Break | auxiliary/GraphemeBreakProperty.txt |
| grapheme GB11 (emoji ZWJ) | Extended_Pictographic | emoji/emoji-data.txt |
| grapheme GB9c (Indic) | Indic_Conjunct_Break | DerivedCoreProperties.txt |
(?i) |
simple case folding | CaseFolding.txt |
\s |
White_Space | PropList.txt |
The data (property tables + the version-dependent GraphemeBreak / Emoji /
Incb enum value lists) is generated. The algorithms that consume it — the
UTF-8 codec and the UAX #29 grapheme break rules (is_grapheme_boundary) — are
hand-maintained C++ in regexlib.h, outside the generated block. Algorithms are
not auto-derived from the spec; only the property data they read is.
The current target is recorded in ucd/VERSION (Unicode 17.0.0) and pinned by
SHA-256 in ucd/SHA256SUMS. The raw *.txt inputs are large and reproducible, so
they are not committed (ucd/.gitignore); fetch them on demand.
python3 tools/fetch_ucd.py # download + verify pinned inputs
python3 tools/generate_ucd.py verify # parse sanity checks
python3 tools/generate_ucd.py update regexlib.h # splice the generated blockThe generated block lives between // [BEGIN GENERATED UCD BLOCK] and
// [END GENERATED UCD BLOCK] in regexlib.h.
- Edit
ucd/VERSION. python3 tools/fetch_ucd.py --refresh-sums(re-pinsSHA256SUMS; review the diff).python3 tools/generate_ucd.py update regexlib.h.- Review the UAX #29 rule skeleton (
is_grapheme_boundaryinregexlib.h) against the new UAX #29 revision history — new rules (rare) are hand-applied; new property values come through automatically via the regenerated enums/tables. - Run the tests, including the grapheme oracle below.
tools/generate_ucd.py verify— parse-level sanity.- Full-range self-check — every regeneration re-derives each property directly from the parsed UCD and asserts the emitted table matches across all code points (General_Category, Grapheme_Cluster_Break, Emoji, White_Space, and simple case folding). Code points that have only a full fold (ß, İ, …) have no simple fold and return the code point unchanged.
- Official UAX #29 test —
ucd/auxiliary/GraphemeBreakTest.txtpasses 766/766 throughreg::unicode::is_grapheme_boundary. - The regexlib test/fuzz/corpus suites pass with the generated tables.
check_doc_identifiers.py— verifies that backtick identifiers indocs/*.md/README.mdstill exist inregexlib.h/test/*.cc, so a rename cannot silently rot the docs. Run asjust lint-docs; CI runs it on every push.