ZMat is a lightweight Python C extension for fast, in-memory data compression and decompression. It provides a unified interface to multiple compression algorithms, all compiled directly into the module with zero external dependencies.

ZMat is part of the NeuroJSON project and is supported by the US National Institute of Health (NIH) grant U24-NS124027.

• GitHub: https://github.com/NeuroJSON/zmat
• Documentation: https://neurojson.org/Page/zmat

pip install zmat

git clone https://github.com/NeuroJSON/zmat.git
cd zmat/python
pip install .

All compression libraries (miniz, easylzma, lz4, zstd, blosc2) are embedded in the source and compiled directly into the module. No system libraries are required.

import zmat

# Compress data
data = b"Hello, ZMat! " * 1000
compressed = zmat.compress(data)
print(f"Original: {len(data)} bytes -> Compressed: {len(compressed)} bytes")

# Decompress data
restored = zmat.decompress(compressed)
assert restored == data

# Use different algorithms
fast = zmat.compress(data, method='lz4')       # fastest
small = zmat.compress(data, method='lzma')      # smallest
balanced = zmat.compress(data, method='zstd')   # good balance

# Base64 encoding/decoding
encoded = zmat.encode(data, method='base64')
decoded = zmat.decode(encoded, method='base64')
assert decoded == data

Compress a bytes-like object.

Parameters:

• data — bytes, bytearray, or any object supporting the buffer protocol
• method — compression algorithm name (default: 'zlib')
• level — compression level: 1 for default, higher values (up to 9 or 12 depending on algorithm) for more compression at the cost of speed

Returns: bytes — compressed data

Decompress a bytes-like object.

Parameters:

• data — bytes or bytearray of compressed data
• method — compression algorithm used to compress the data (default: 'zlib')

Returns: bytes — decompressed data

Encode data (e.g., base64 encoding).

Parameters:

• data — bytes or bytearray to encode
• method — encoding method (default: 'base64')

Returns: bytes — encoded data

Decode data (e.g., base64 decoding).

Parameters:

• data — bytes or bytearray of encoded data
• method — encoding method used (default: 'base64')

Returns: bytes — decoded data

Low-level compression/decompression interface with full control over all parameters, including blosc2 multi-threading options.

Parameters:

• data — bytes, bytearray, or buffer-protocol object
• iscompress — 1 to compress (default level), 0 to decompress, negative values to set compression level (e.g., -9 for maximum)
• method — algorithm name
• nthread — number of threads for blosc2 (default: 1)
• shuffle — byte shuffle for blosc2: 0 disabled, 1 enabled (default: 1)
• typesize — element byte size for blosc2 shuffle (default: 4)

Returns: bytes — compressed or decompressed data

ZMat's zlib and gzip output is fully compatible with Python's standard library:

import zmat
import zlib
import gzip
import io

data = b"interoperability test " * 100

# zmat -> Python zlib
compressed = zmat.compress(data, method='zlib')
assert zlib.decompress(compressed) == data

# Python zlib -> zmat
compressed = zlib.compress(data)
assert zmat.decompress(compressed, method='zlib') == data

# zmat -> Python gzip
compressed = zmat.compress(data, method='gzip')
with gzip.open(io.BytesIO(compressed), 'rb') as f:
    assert f.read() == data

ZMat accepts any object supporting Python's buffer protocol, including NumPy arrays:

import numpy as np
import zmat

arr = np.random.rand(1000, 1000)
compressed = zmat.compress(arr.tobytes(), method='lz4')
restored = np.frombuffer(zmat.decompress(compressed, method='lz4'),
                         dtype=arr.dtype).reshape(arr.shape)
assert np.array_equal(arr, restored)

For numerical arrays, the blosc2 methods with byte-shuffle can achieve better compression ratios:

compressed = zmat.zmat(arr.tobytes(), iscompress=1, method='blosc2zstd',
                       typesize=8)  # 8 bytes per float64 element

The build can be customized via environment variables:

Example: build without blosc2 for a smaller binary:

ZMAT_NO_BLOSC2=1 pip install .

ZMat is also available as a MATLAB/Octave MEX function with an identical feature set. See the main README for details on the MATLAB/Octave toolbox.

ZMat bundles the following open-source libraries:

• zlib/miniz — Jean-loup Gailly, Mark Adler / Rich Geldreich (MIT/Zlib license)
• easylzma — Lloyd Hilaiel (public domain)
• LZMA SDK — Igor Pavlov (public domain)
• LZ4 — Yann Collet (BSD 2-Clause)
• Zstandard — Meta Platforms, Inc. (BSD 3-Clause)
• C-Blosc2 — Blosc Development Team (BSD 3-Clause)
• base64 — Jouni Malinen (BSD license)

ZMat is licensed under the GNU General Public License v3.

Copyright (C) 2019-2026 Qianqian Fang <<q.fang at neu.edu>>