Better fastbase64 + man pages by lemire · Pull Request #947 · simdutf/simdutf

lemire · 2026-03-13T21:44:18Z

This makes fastbase64 somewhat equivalent to standard base64. Unfortunately, the GNU base64 and the non-GNU base64 tools differ. It tried to come up with some kind of hybrid that can emulate the two.

Apple M4 Benchmark Results (times in milliseconds):

Size	Encode Base64	Encode FastBase64	Decode Base64	Decode FastBase64
1m	23.5	23.2	37.7	23.1
10m	31.3	24.4	167.6	24.0
100m	115.6	50.4	1430.2	53.7

x64 Linux Benchmark Results (times in milliseconds):

Size	Encode Base64	Encode FastBase64	Decode Base64	Decode FastBase64
1m	12.1	11.8	13.8	11.8
10m	25.8	18.7	39.6	17.5
100m	187.0	100.5	290.3	77.7

So for inputs that are 10 MB or larger, it pays to use fastbase64. For small inputs, the time is dominated by the process overhead.

The Apple base64 decoder is quite slow, so that's one use case where we may want to use fastbase64.

Fixes #907

Fixes #908

tools/fastbase64.cpp

scripts/base64bench.sh

pauldreik · 2026-03-15T09:16:56Z

I think it would be great to provide a drop in replacement for gnu coreutils.
We could provide a bash script wrapper fastbase64.coreutilscompat that parses command line parameters and uses defaults that exactly matches gnu coreutils.

then, it would be possible for users to either use an alias in .bashrc to get some nice speedup.

the same thing could be done for mac/bsd, I assume?

lemire · 2026-03-15T18:58:11Z

@pauldreik

I think it would be great to provide a drop in replacement for gnu coreutils.

I think that the only difference is the meaning of the flag -i.

We have a few options.

Add an intermediate script as you suggest.
Generate two binaries, one with GNU commands, one with BSD convention.
Focus solely on the GNU convention.
Stick with what I did which is mostly cross-compatible except that Linux users must not use -i.
Detect a GNU (Linux) system and change the behaviour under Linux so that -i means ignore bad characters.

lemire · 2026-03-22T14:35:02Z

I am asking for advice on X : https://x.com/lemire/status/2035726774919713271?s=61&t=v2gDAuOzz1C3ICrzUYCSlQ

lemire · 2026-03-22T14:53:49Z

This answer is interesting: https://x.com/austinsnotweird/status/2035730879461658876?s=61&t=v2gDAuOzz1C3ICrzUYCSlQ

chadbrewbaker · 2026-03-22T15:01:12Z

I would defer to @sylvestre. The last Rust coreutils PR was for arg[0] handling: uutils/coreutils@6c3de3e

--bsdmode, --gnumode seem like the most sensible flags for a utility that isn't GNU or BSD?

pauldreik · 2026-03-22T19:49:51Z

I think busybox is also using the trick of checking argv[0] to let the same binary masquerade as different programs.

In the case of providing a symlink with a name of say fastbase64.coreutils which links to fastbase64, argv[0] is "/path/to/fastbase64.coreutils" (depends on how the link is invoked, one needs to do basename on it)

I think some people use gnu coreutils on mac, so it makes sense to provide the .coreutils link not only on linux.

I made a quick test of trying to make a symlink with cmake, using:

install(CODE "execute_process( \
    COMMAND echo ${CMAKE_COMMAND} -E create_symlink \
    ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/fastbase64 \
    ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/fastbase64.coreutils   \
    )"
)

it does not work properly, I think a generator expression would have been ideal but is does not seem like it is supported: https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#export-and-install-expressions

there is a potentially useful hint here but I failed understanding how: https://discourse.cmake.org/t/how-to-properly-install-a-symbolic-link-for-backwards-compatibility/8592

lemire · 2026-03-23T20:49:12Z

@pauldreik Please see

#949

I just call ln directly or copy the file as a fallback when ln is not available.

Now, I wonder about the name. Is fastbase64.coreutils what we want to go with?

* This creates two fastbase64 utilities. We have fastbase64 that is meant as a dropin replacement for the standard BSD/mac util. And fastbase.coreutils which is meant as a replacement for the GNU coreutils base64 program. * minor fix for windows * minor fixes * minor fix * another option? * Update man/fastbase64.coreutils.1 Co-authored-by: Copilot <[email protected]> * Update tools/CMakeLists.txt Co-authored-by: Copilot <[email protected]> * handling gnumode differently. * Update man/fastbase64.coreutils.1 Co-authored-by: Copilot <[email protected]> * Update man/fastbase64.coreutils.1 Co-authored-by: Copilot <[email protected]> * putting back the damn thing. * another fix * Update tools/CMakeLists.txt Co-authored-by: Copilot <[email protected]> * tuning. * adding tests * more testing * fixing windows * make script more portable * more windows fixes * Update tests/CMakeLists.txt Co-authored-by: Copilot <[email protected]> * Update tests/fastbase64/test_fastbase64.py Co-authored-by: Copilot <[email protected]> * Update tests/fastbase64/test_fastbase64.py Co-authored-by: Copilot <[email protected]> * various additional fixes * Update tools/fastbase64.cpp Co-authored-by: Copilot <[email protected]> * Update man/fastbase64.1 Co-authored-by: Copilot <[email protected]> * Update man/fastbase64.coreutils.1 Co-authored-by: Copilot <[email protected]> * Update tools/CMakeLists.txt Co-authored-by: Copilot <[email protected]> * Update tools/fastbase64.cpp Co-authored-by: Copilot <[email protected]> * tweaks * forgot to check it in. * many more fixes * more tweaking * more fixes * Update README.md Co-authored-by: Copilot <[email protected]> * Update tests/fastbase64/test_fastbase64.py Co-authored-by: Copilot <[email protected]> * more fixing. * minor fix * more tests * tuning. * ok * making it prettier. * more fine tuning. --------- Co-authored-by: Copilot <[email protected]>

lemire · 2026-03-25T00:38:07Z

@pauldreik Ok, now the PR does have two utilities.

Co-authored-by: Copilot <[email protected]>

…ools

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tools/fastbase64.cpp

tests/fastbase64/test_fastbase64.py

lemire · 2026-03-25T01:19:23Z

@pauldreik This should be in pretty decent shape now.

pauldreik

I like the added functionality!

when I run the test, I get failures:

tests/fastbase64/test_fastbase64.py build/release/tools/fastbase64 build/release/tools/fastbase64.coreutils recovered_base64_100m.bin 

=== fastbase64 (BSD-like) ===
 PASS --help
 PASS --version
 PASS unknown option rejected
 PASS -n rejected in BSD mode
 FAIL round-trip: -e FILE | -d
 PASS encoded matches expected
 PASS -e / -d explicit flags (binary data)
 PASS --encode / --decode long options
 PASS -D decode alias
 PASS -b 76 wraps at 76 chars
 PASS -w 40 wraps at 40 chars
 PASS --wrap=60 wraps at 60 chars
 PASS default no-wrap: no interior newlines
 PASS base64 alphabet in output
 PASS empty input round-trip
 PASS single-byte round-trip
 PASS all-256-byte-values round-trip
 FAIL -i FILE (explicit input file)
 PASS --input FILE
 FAIL -o FILE (explicit output file)
 FAIL --output FILE
 PASS second positional arg as output file
 PASS invalid base64 input fails
 PASS '-' reads from stdin
 FAIL wrapped encode round-trips correctly
 PASS fastbase64: corrupt input fails without ignore-garbage
 FAIL fastbase64: --ignore-garbage recovers

=== fastbase64.coreutils (GNU-like) ===
 PASS --help
 PASS --version
 PASS unknown option rejected
 FAIL round-trip
 PASS -e / -d explicit flags (binary data)
 PASS --encode / --decode long options
 PASS -D decode alias
 PASS default wrapping: has interior newlines
 PASS base64 alphabet in output
 PASS -w 76 wraps at 76 chars
 PASS -w 0 no-wrap: no interior newlines
 PASS --wrap=40 wraps at 40 chars
 PASS -b 76 is alias for -w 76
 PASS empty input round-trip
 PASS all-256-byte-values round-trip
 PASS corrupt input fails without ignore-garbage
 FAIL -i ignores garbage and decodes correctly
 FAIL --ignore-garbage long option
 FAIL -n ignores garbage and decodes correctly
 FAIL --noerrcheck long option
 PASS -o FILE output
 PASS second positional arg as output file
 FAIL '-' reads from stdin
 FAIL wrapped encode round-trips correctly

=== Large inputs round-trips (over 64K, deterministic) ===
 PASS fastbase64 round-trip 65,536 bytes
 PASS coreutils round-trip 65,536 bytes
 PASS fastbase64 wrapped 65,536 bytes
 PASS coreutils wrapped 65,536 bytes
 PASS large alphabet check 65,536 bytes
 PASS fastbase64 round-trip 65,537 bytes
 PASS coreutils round-trip 65,537 bytes
 PASS fastbase64 wrapped 65,537 bytes
 PASS coreutils wrapped 65,537 bytes
 PASS large alphabet check 65,537 bytes
 PASS fastbase64 round-trip 131,072 bytes
 PASS coreutils round-trip 131,072 bytes
 PASS fastbase64 wrapped 131,072 bytes
 PASS coreutils wrapped 131,072 bytes
 PASS large alphabet check 131,072 bytes
 PASS fastbase64 round-trip 262,144 bytes
 PASS coreutils round-trip 262,144 bytes
 PASS fastbase64 wrapped 262,144 bytes
 PASS coreutils wrapped 262,144 bytes
 PASS large alphabet check 262,144 bytes
 PASS fastbase64 round-trip 524,288 bytes
 PASS coreutils round-trip 524,288 bytes
 PASS fastbase64 wrapped 524,288 bytes
 PASS coreutils wrapped 524,288 bytes
 PASS large alphabet check 524,288 bytes
 PASS fastbase64 round-trip 1,048,576 bytes
 PASS coreutils round-trip 1,048,576 bytes
 PASS fastbase64 wrapped 1,048,576 bytes
 PASS coreutils wrapped 1,048,576 bytes
 PASS large alphabet check 1,048,576 bytes
 PASS fastbase64 round-trip 2,097,152 bytes
 PASS coreutils round-trip 2,097,152 bytes
 PASS fastbase64 wrapped 2,097,152 bytes
 PASS coreutils wrapped 2,097,152 bytes
 PASS large alphabet check 2,097,152 bytes

=== Wrapping extremes & edge options ===
 PASS fastbase64 -w 1 round-trip
 PASS fastbase64 -w 1 line lengths correct
 PASS fastbase64 -w 4 round-trip
 PASS fastbase64 -w 4 line lengths correct
 PASS fastbase64 -w 76 round-trip
 PASS fastbase64 -w 76 line lengths correct
 PASS fastbase64 -w 100000 round-trip
 PASS fastbase64 -w 100000 line lengths correct
 PASS coreutils -w 1 round-trip
 PASS coreutils -w 1 line lengths correct
 PASS coreutils -w 4 round-trip
 PASS coreutils -w 4 line lengths correct
 PASS coreutils -w 76 round-trip
 PASS coreutils -w 76 line lengths correct
 PASS coreutils -w 100000 round-trip
 PASS coreutils -w 100000 line lengths correct

=== Adversarial decoding (padding, garbage, whitespace) ===
 PASS build/release/tools/fastbase64 rejects garbage without ignore
 PASS build/release/tools/fastbase64 ignore-garbage recovers
 PASS build/release/tools/fastbase64.coreutils rejects garbage without ignore
 PASS build/release/tools/fastbase64.coreutils ignore-garbage recovers
 PASS build/release/tools/fastbase64 rejects garbage without ignore
 PASS build/release/tools/fastbase64 ignore-garbage recovers
 PASS build/release/tools/fastbase64.coreutils rejects garbage without ignore
 PASS build/release/tools/fastbase64.coreutils ignore-garbage recovers
 PASS build/release/tools/fastbase64 handles 100KB single-line base64
 PASS build/release/tools/fastbase64.coreutils handles 100KB single-line base64

=== Chunk-boundary whitespace stress ===
 PASS fastbase64: space at chunk boundary
 PASS fastbase64: space one before boundary
 PASS fastbase64: space one after boundary
 PASS fastbase64: newline at chunk boundary
 PASS fastbase64: 10 spaces at boundary
 PASS fastbase64: CRLF at boundary
 PASS fastbase64: spaces straddling boundary -3..+3
 PASS fastbase64: tab at boundary
 PASS fastbase64: mixed ws at boundary
 PASS fastbase64: 50 newlines at boundary
 PASS fastbase64: space at 2×chunk boundary
 PASS fastbase64: newline at 2×chunk boundary
 PASS fastbase64: 10 spaces at 2×chunk boundary
 PASS fastbase64: spaces near 2×chunk -5..+5
 PASS fastbase64: spaces at both chunk boundaries
 PASS fastbase64: space at group boundary before chunk
 PASS fastbase64: space at group boundary after chunk
 PASS fastbase64: 1000 spaces at boundary
 PASS fastbase64: newline every 76 chars
 PASS fastbase64: trailing spaces before final newline
 PASS coreutils: space at chunk boundary
 PASS coreutils: space one before boundary
 PASS coreutils: space one after boundary
 PASS coreutils: newline at chunk boundary
 PASS coreutils: 10 spaces at boundary
 PASS coreutils: CRLF at boundary
 PASS coreutils: spaces straddling boundary -3..+3
 PASS coreutils: tab at boundary
 PASS coreutils: mixed ws at boundary
 PASS coreutils: 50 newlines at boundary
 PASS coreutils: space at 2×chunk boundary
 PASS coreutils: newline at 2×chunk boundary
 PASS coreutils: 10 spaces at 2×chunk boundary
 PASS coreutils: spaces near 2×chunk -5..+5
 PASS coreutils: spaces at both chunk boundaries
 PASS coreutils: space at group boundary before chunk
 PASS coreutils: space at group boundary after chunk
 PASS coreutils: 1000 spaces at boundary
 PASS coreutils: newline every 76 chars
 PASS coreutils: trailing spaces before final newline

=== Super sparse base64 decoding ===
 PASS fastbase64: 0 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS coreutils: 0 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS fastbase64: 1 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS coreutils: 1 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS fastbase64: 2 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS coreutils: 2 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS fastbase64: 3 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly
 PASS coreutils: 3 valid payloads in 197608 bytes (spanning 4 chunks) decoded correctly

=== Error conditions ===
 PASS fastbase64: non-existent input file rejected
 PASS fastbase64: bad -w -1 rejected
 PASS fastbase64: bad -w abc rejected
 PASS fastbase64: bad -w 999999999999999 rejected
 PASS coreutils: non-existent input file rejected
 PASS coreutils: bad -w -1 rejected
 PASS coreutils: bad -w abc rejected
 PASS coreutils: bad -w 999999999999999 rejected

============================================================
Results: 155 passed, 13 failed

man/fastbase64.coreutils.1

tools/fastbase64.cpp

lemire · 2026-03-29T23:04:45Z

@pauldreik The problem was with the python script which I revised.

lemire added 4 commits March 13, 2026 16:33

better tools

97fe62a

adding script.

315ddc1

update

e739944

don't be so silly

d4dfd38

lemire requested a review from pauldreik March 13, 2026 21:44

lint

a76f15b

lemire mentioned this pull request Mar 13, 2026

make fastbase64 functionally equivalent to the standard base64 command #908

Open

lemire added 2 commits March 14, 2026 17:50

updating script

5209e7f

update results.

945b7ce

pauldreik reviewed Mar 15, 2026

View reviewed changes

lemire added 5 commits March 15, 2026 13:50

various minor fixes

2ec1de8

using std::fputc

7ffaca6

various fixes

9ebc1d1

more fixes.

c727ec7

fixing script

5b00340

lemire requested review from Copilot and pauldreik March 25, 2026 00:38

This comment was marked as resolved.

Sign in to view

Copilot started reviewing on behalf of lemire March 25, 2026 00:48 View session

lemire and others added 2 commits March 24, 2026 20:52

Update tests/CMakeLists.txt

e153119

Co-authored-by: Copilot <[email protected]>

Update tools/CMakeLists.txt

f28b30f

Co-authored-by: Copilot <[email protected]>

lemire added 3 commits March 24, 2026 20:52

a few fixes

c53023e

Merge branch 'bettertools' of github.com:simdutf/simdutf into bettert…

9366b34

…ools

adding see also.

263edbc

lemire requested a review from Copilot March 25, 2026 00:59

fixing bad exceptions.

7ba1f64

Copilot AI reviewed Mar 25, 2026

View reviewed changes

tools/fastbase64.cpp Show resolved Hide resolved

tools/fastbase64.cpp Show resolved Hide resolved

tests/fastbase64/test_fastbase64.py Show resolved Hide resolved

lint

c69d0b5

Copilot started reviewing on behalf of lemire March 25, 2026 02:46 View session

pauldreik reviewed Mar 25, 2026

View reviewed changes

lemire added 5 commits March 27, 2026 11:51

minor tweaks.

0a991f8

fixing script.

35b6fde

fixing test script.

a9e9c26

Merge branch 'master' into bettertools

553bb64

Merge branch 'master' into bettertools

e278e9f

lemire requested a review from pauldreik March 29, 2026 23:04

lemire added 2 commits March 29, 2026 20:55

lint

a57c78d

Merge branch 'master' into bettertools

0c0bbd0

Conversation

lemire commented Mar 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pauldreik commented Mar 15, 2026

Uh oh!

lemire commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lemire commented Mar 22, 2026

Uh oh!

lemire commented Mar 22, 2026

Uh oh!

chadbrewbaker commented Mar 22, 2026

Uh oh!

pauldreik commented Mar 22, 2026

Uh oh!

lemire commented Mar 23, 2026

Uh oh!

lemire commented Mar 25, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lemire commented Mar 25, 2026

Uh oh!

pauldreik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lemire commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lemire commented Mar 15, 2026 •

edited

Loading