MemoryPack

Zero-encoding, source-generated binary serializer for modern .NET.

Compared with System.Text.Json, protobuf-net, MessagePack for C#, Orleans.Serialization. These serializers have IBufferWriter<byte> methods and can reuse buffers to avoid measuring output copies.

For standard objects, MemoryPack is x10 faster and x2 ~ x5 faster than other binary serializers. For struct array, MemoryPack is even more powerful, with speeds up to x50 ~ x200 greater than other serializers.

MemoryPack is my 4th serializer, previously I've created well known serializers, ZeroFormatter, Utf8Json, MessagePack for C#. Its speed comes from a C#-specific binary format, modern .NET APIs and an incremental source generator.

Other serializers perform many encoding operations such as VarInt encoding, tag, string, etc. MemoryPack format uses a zero-encoding design that copies as much C# memory as possible. Zero-encoding is similar to FlatBuffers, but it doesn't need a special type, MemoryPack's serialization target is POCO.

Other than performance, MemoryPack has these features.

• Support modern I/O APIs (IBufferWriter<byte>, ReadOnlySpan<byte>, ReadOnlySequence<byte>)
• Native AOT friendly Source Generator based code generation, no Dynamic CodeGen (IL.Emit)
• Deserialize into existing instance
• Polymorphism (Union) serialization
• Limited version-tolerant (fast/default) and full version-tolerant support
• Circular reference serialization
• PipeWriter/Reader based streaming serialization
• TypeScript code generation
• Explicit formatter contexts for collectible AssemblyLoadContext and RPC graphs

This branch targets .NET 10 and uses C# 14.

PM> Install-Package MemoryPack

The runtime packages target net10.0. MemoryPack.Generator targets netstandard2.0 only because Roslyn analyzers are loaded by the compiler host; that target does not add a netstandard runtime support promise.

Define a struct or class to be serialized and annotate it with the [MemoryPackable] attribute and the partial keyword.

using MemoryPack;

[MemoryPackable]
public partial class Person
{
    public int Age { get; set; }
    public string Name { get; set; }
}

Serialization code is generated by the C# source generator feature which implements the IMemoryPackable<T> interface. In Visual Studio you can check a generated code by using a shortcut Ctrl+K, R on the class name and select *.MemoryPackFormatter.g.cs.

Call MemoryPackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize an object instance.

var v = new Person { Age = 40, Name = "John" };

var bin = MemoryPackSerializer.Serialize(v);
var val = MemoryPackSerializer.Deserialize<Person>(bin);

Serialize supports byte[], IBufferWriter<byte> and Stream. Deserialize supports ReadOnlySpan<byte>, ReadOnlySequence<byte> and Stream. Serializer operations are generic-only.

The parameterless serializer APIs use process-lifetime generic formatter slots directly; there is no global default-context object or mutable formatter graph. This is the fastest path and is appropriate for application types loaded into the default AssemblyLoadContext. Using these APIs with a type from a collectible AssemblyLoadContext does not provide an unload guarantee.

Use an explicit MemoryPackSerializerContext when formatter lifetime must follow a plugin or an RPC endpoint:

var context = new MemoryPackSerializerContext();

var bin = MemoryPackSerializer.Serialize(pluginValue, context);
var value = MemoryPackSerializer.Deserialize<PluginDto>(bin, context);

The same context is propagated through the complete object graph, including arrays, lists, dictionaries, generated types, unions and custom formatters. Collectible types and registered overrides always use the context-owned graph; an empty/configuration-only context may share type-only static slots for non-collectible types.

Build a frozen context when custom formatters or UTF-16 output are needed:

var context = new MemoryPackSerializerContextBuilder()
    .Configure(MemoryPackSerializerConfiguration.Utf16)
    .Register<IPluginContract>(new PluginContractFormatter())
    .Build();

There is no mutable global formatter provider. Release the context together with plugin values, assemblies, reflection objects and delegates before unloading the plugin AssemblyLoadContext.

See Formatter contexts and collectible AssemblyLoadContext for lifecycle rules, migration details, architecture, and benchmark results.

These types can be serialized by default:

• .NET primitives (byte, int, bool, char, double, etc.)
• Unmanaged types (Any enum, Any user-defined struct which doesn't contain reference types)
• string, decimal, Half, Int128, UInt128, Guid, Rune, BigInteger
• TimeSpan, DateTime, DateTimeOffset, TimeOnly, DateOnly, TimeZoneInfo
• Complex, Plane, Quaternion Matrix3x2, Matrix4x4, Vector2, Vector3, Vector4
• Uri, Version, StringBuilder, Type, BitArray, CultureInfo
• T[], T[,], T[,,], T[,,,], Memory<>, ReadOnlyMemory<>, ArraySegment<>, ReadOnlySequence<>
• Nullable<>, Lazy<>, KeyValuePair<,>, Tuple<,...>, ValueTuple<,...>
• List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, SortedSet<>, PriorityQueue<,>
• Dictionary<,>, SortedList<,>, SortedDictionary<,>, ReadOnlyDictionary<,>
• Collection<>, ReadOnlyCollection<>, ObservableCollection<>, ReadOnlyObservableCollection<>
• IEnumerable<>, ICollection<>, IList<>, IReadOnlyCollection<>, IReadOnlyList<>, ISet<>
• IDictionary<,>, IReadOnlyDictionary<,>, ILookup<,>, IGrouping<,>,
• ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary<,>, BlockingCollection<>
• Immutable collections (ImmutableList<>, etc.) and interfaces (IImmutableList<>, etc.)

[MemoryPackable] can annotate to any class, struct, record, record struct and interface. If a type is struct or record struct which contains no reference types (C# Unmanaged types) any additional annotation (ignore, include, constructor, callbacks) is not used, that serialize/deserialize directly from the memory.

Otherwise, by default, [MemoryPackable] serializes public instance properties or fields. You can use [MemoryPackIgnore] to remove serialization target, [MemoryPackInclude] promotes a private member to serialization target.

[MemoryPackable]
public partial class Sample
{
    // these types are serialized by default
    public int PublicField;
    public readonly int PublicReadOnlyField;
    public int PublicProperty { get; set; }
    public int PrivateSetPublicProperty { get; private set; }
    public int ReadOnlyPublicProperty { get; }
    public int InitProperty { get; init; }
    public required int RequiredInitProperty { get; init; }

    // these types are not serialized by default
    int privateProperty { get; set; }
    int privateField;
    readonly int privateReadOnlyField;

    // use [MemoryPackIgnore] to remove target of a public member
    [MemoryPackIgnore]
    public int PublicProperty2 => PublicProperty + PublicField;

    // use [MemoryPackInclude] to promote a private member to serialization target
    [MemoryPackInclude]
    int privateField2;
    [MemoryPackInclude]
    int privateProperty2 { get; set; }
}

MemoryPack's code generator adds information about what members are serialized to the <remarks /> section. This can be viewed by hovering over the type with Intellisense.

All members must be memorypack-serializable, if not the code generator will emit an error.

MemoryPack has 35 diagnostics rules (MEMPACK001 to MEMPACK035) to be defined comfortably.

If target type is defined MemoryPack serialization externally and registered, use [MemoryPackAllowSerialize] to silent diagnostics.

[MemoryPackable]
public partial class Sample2
{
    [MemoryPackAllowSerialize]
    public NotSerializableType? NotSerializableProperty { get; set; }
}

Member order is important, MemoryPack does not serialize the member-name or other information, instead serializing fields in the order they are declared. If a type is inherited, serialization is performed in the order of parent → child. The order of members can not change for the deserialization. For the schema evolution, see the Version tolerant section.

The default order is sequential, but you can choose the explicit layout with [MemoryPackable(SerializeLayout.Explicit)] and [MemoryPackOrder()].

// serialize Prop0 -> Prop1
[MemoryPackable(SerializeLayout.Explicit)]
public partial class SampleExplicitOrder
{
    [MemoryPackOrder(1)]
    public int Prop1 { get; set; }
    [MemoryPackOrder(0)]
    public int Prop0 { get; set; }
}

MemoryPack supports both parameterized and parameterless constructors. The selection of the constructor follows these rules. (Applies to classes and structs).

• If there is [MemoryPackConstructor], use it.
• If there is no explicit constructor (including private), use a parameterless one.
• If there is one parameterless/parameterized constructor (including private), use it.
• If there are multiple constructors, then the [MemoryPackConstructor] attribute must be applied to the desired constructor (the generator will not automatically choose one), otherwise the generator will emit an error.
• If using a parameterized constructor, all parameter names must match corresponding member names (case-insensitive).

[MemoryPackable]
public partial class Person
{
    public readonly int Age;
    public readonly string Name;

    // You can use a parameterized constructor - parameter names must match corresponding members name (case-insensitive)
    public Person(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

// also supports record primary constructor
[MemoryPackable]
public partial record Person2(int Age, string Name);

public partial class Person3
{
    public int Age { get; set; }
    public string Name { get; set; }

    public Person3()
    {
    }

    // If there are multiple constructors, then [MemoryPackConstructor] should be used
    [MemoryPackConstructor]
    public Person3(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

When serializing/deserializing, MemoryPack can invoke a before/after event using the [MemoryPackOnSerializing], [MemoryPackOnSerialized], [MemoryPackOnDeserializing], [MemoryPackOnDeserialized] attributes. It can annotate both static and instance (non-static) methods, and public and private methods.

[MemoryPackable]
public partial class MethodCallSample
{
    // method call order is static -> instance
    [MemoryPackOnSerializing]
    public static void OnSerializing1()
    {
        Console.WriteLine(nameof(OnSerializing1));
    }

    // also allows private method
    [MemoryPackOnSerializing]
    void OnSerializing2()
    {
        Console.WriteLine(nameof(OnSerializing2));
    }

    // serializing -> /* serialize */ -> serialized
    [MemoryPackOnSerialized]
    static void OnSerialized1()
    {
        Console.WriteLine(nameof(OnSerialized1));
    }

    [MemoryPackOnSerialized]
    public void OnSerialized2()
    {
        Console.WriteLine(nameof(OnSerialized2));
    }

    [MemoryPackOnDeserializing]
    public static void OnDeserializing1()
    {
        Console.WriteLine(nameof(OnDeserializing1));
    }

    // Note: instance method with MemoryPackOnDeserializing, that not called if instance is not passed by `ref`
    [MemoryPackOnDeserializing]
    public void OnDeserializing2()
    {
        Console.WriteLine(nameof(OnDeserializing2));
    }

    [MemoryPackOnDeserialized]
    public static void OnDeserialized1()
    {
        Console.WriteLine(nameof(OnDeserialized1));
    }

    [MemoryPackOnDeserialized]
    public void OnDeserialized2()
    {
        Console.WriteLine(nameof(OnDeserialized2));
    }
}

Callbacks allows parameterless method and ref reader/writer, ref T value method. For example, ref callbacks can write/read custom header before serialization process.

[MemoryPackable]
public partial class EmitIdData
{
    public int MyProperty { get; set; }

    [MemoryPackOnSerializing]
    static void WriteId<TBufferWriter>(ref MemoryPackWriter<TBufferWriter> writer, ref EmitIdData? value)
        where TBufferWriter : IBufferWriter<byte> // .NET Standard 2.1, use where TBufferWriter : class, IBufferWriter<byte>
    {
        writer.WriteUnmanaged(Guid.NewGuid()); // emit GUID in header.
    }

    [MemoryPackOnDeserializing]
    static void ReadId(ref MemoryPackReader reader, ref EmitIdData? value)
    {
        // read custom header before deserialize
        var guid = reader.ReadUnmanaged<Guid>();
        Console.WriteLine(guid);
    }
}

Implement MemoryPackFormatter<T> and register the closed formatter while building a context:

var context = new MemoryPackSerializerContextBuilder()
    .Register<ExternalType>(new ExternalTypeFormatter())
    .Build();

Registrations are frozen by Build and are isolated to that context. A generated wrapper type remains the simplest option when you control the RPC contract.

• MemoryPack — runtime plus source generator
• MemoryPack.Core — runtime primitives and formatter graph
• MemoryPack.Generator — Roslyn incremental source generator
• MemoryPack.Streaming — PipeReader/PipeWriter and streaming APIs

MemoryPack supports TypeScript code generation. It generates class and serialization code from C#, In other words, you can share types with the Browser without using OpenAPI, proto, etc.

Code generation is integrated with Source Generator, the following options(MemoryPackGenerator_TypeScriptOutputDirectory) set the output directory for TypeScript code. Runtime code is output at the same time, so no additional dependencies are required.

<!-- output memorypack TypeScript code to directory -->
<ItemGroup>
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptOutputDirectory" />
</ItemGroup>
<PropertyGroup>
    <MemoryPackGenerator_TypeScriptOutputDirectory>$(MSBuildProjectDirectory)\wwwroot\js\memorypack</MemoryPackGenerator_TypeScriptOutputDirectory>
</PropertyGroup>

A C# MemoryPackable type must be annotated with [GenerateTypeScript].

[MemoryPackable]
[GenerateTypeScript]
public partial class Person
{
    public required Guid Id { get; init; }
    public required int Age { get; init; }
    public required string FirstName { get; init; }
    public required string LastName { get; init; }
    public required DateTime DateOfBirth { get; init; }
    public required Gender Gender { get; init; }
    public required string[] Emails { get; init; }
}

public enum Gender
{
    Male, Female, Other
}

Runtime code and TypeScript type will be generated in the target directory.

The generated code is as follows, with simple fields and static methods for serialize/serializeArray and deserialize/deserializeArray.

import { MemoryPackWriter } from "./MemoryPackWriter.js";
import { MemoryPackReader } from "./MemoryPackReader.js";
import { Gender } from "./Gender.js"; 

export class Person {
    id: string;
    age: number;
    firstName: string | null;
    lastName: string | null;
    dateOfBirth: Date;
    gender: Gender;
    emails: (string | null)[] | null;

    constructor() {
        // snip...
    }

    static serialize(value: Person | null): Uint8Array {
        // snip...
    }

    static serializeCore(writer: MemoryPackWriter, value: Person | null): void {
        // snip...
    }

    static serializeArray(value: (Person | null)[] | null): Uint8Array {
        // snip...
    }

    static serializeArrayCore(writer: MemoryPackWriter, value: (Person | null)[] | null): void {
        // snip...
    }
    static deserialize(buffer: ArrayBuffer): Person | null {
        // snip...
    }

    static deserializeCore(reader: MemoryPackReader): Person | null {
        // snip...
    }

    static deserializeArray(buffer: ArrayBuffer): (Person | null)[] | null {
        // snip...
    }

    static deserializeArrayCore(reader: MemoryPackReader): (Person | null)[] | null {
        // snip...
    }
}

You can use this type like following.

let person = new Person();
person.id = crypto.randomUUID();
person.age = 30;
person.firstName = "foo";
person.lastName = "bar";
person.dateOfBirth = new Date(1999, 12, 31, 0, 0, 0);
person.gender = Gender.Other;
person.emails = ["[email protected]", "[email protected]"];

// serialize to Uint8Array
let bin = Person.serialize(person);

let blob = new Blob([bin.buffer], { type: "application/x-memorypack" })

let response = await fetch("http://localhost:5260/api",
    { method: "POST", body: blob, headers: { "Content-Type": "application/x-memorypack" } });

let buffer = await response.arrayBuffer();

// deserialize from ArrayBuffer 
let person2 = Person.deserialize(buffer);

There are a few restrictions on the types that can be generated. Among the primitives, char and decimal are not supported. Also, OpenGenerics type cannot be used.

[GenerateTypeScript] can only be applied to classes and is currently not supported by struct.

In default, MemoryPack generates file extension as .js like import { MemoryPackWriter } from "./MemoryPackWriter.js";. If you want to change other extension or empty, use MemoryPackGenerator_TypeScriptImportExtension to configure it. Also the member name is automatically converted to camelCase. If you want to use original name, use MemoryPackGenerator_TypeScriptConvertPropertyName to false.

<ItemGroup>
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptOutputDirectory" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptImportExtension" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptConvertPropertyName" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptEnableNullableTypes" />
</ItemGroup>
<PropertyGroup>
    <MemoryPackGenerator_TypeScriptOutputDirectory>$(MSBuildProjectDirectory)\wwwroot\js\memorypack</MemoryPackGenerator_TypeScriptOutputDirectory>
    <!-- allows empty -->
    <MemoryPackGenerator_TypeScriptImportExtension></MemoryPackGenerator_TypeScriptImportExtension>
    <!-- default is true -->
    <MemoryPackGenerator_TypeScriptConvertPropertyName>false</MemoryPackGenerator_TypeScriptConvertPropertyName>
    <!-- default is false -->
    <MemoryPackGenerator_TypeScriptEnableNullableTypes>true</MemoryPackGenerator_TypeScriptEnableNullableTypes>
</PropertyGroup>

MemoryPackGenerator_TypeScriptEnableNullableTypes allows C# nullable annotations to be reflected in TypeScript code. The default is false, making everything nullable.

MemoryPack.Streaming provides MemoryPackStreamingSerializer, which adds additional support for serializing and deserializing collections with streams.

public static class MemoryPackStreamingSerializer
{
    public static async ValueTask SerializeAsync<T>(PipeWriter pipeWriter, int count, IEnumerable<T> source, int flushRate = 4096, CancellationToken cancellationToken = default)
    public static async ValueTask SerializeAsync<T>(Stream stream, int count, IEnumerable<T> source, int flushRate = 4096, CancellationToken cancellationToken = default)
    public static async IAsyncEnumerable<T?> DeserializeAsync<T>(PipeReader pipeReader, int bufferAtLeast = 4096, int readMinimumSize = 8192, [EnumeratorCancellation] CancellationToken cancellationToken = default)
    public static IAsyncEnumerable<T?> DeserializeAsync<T>(Stream stream, int bufferAtLeast = 4096, int readMinimumSize = 8192, CancellationToken cancellationToken = default)
}

Implement MemoryPackFormatter<T> and register it through MemoryPackSerializerContextBuilder. There is no process-wide formatter provider.

var context = new MemoryPackSerializerContextBuilder()
    .Register<Skelton>(new SkeltonFormatter())
    .Build();

MemoryPackWriterOptionalStatePool.Rent(context) and MemoryPackReaderOptionalStatePool.Rent(context) are available when using the low-level reader/writer API directly.

For length-prefixed transports, MemoryPack.Streaming exposes SerializeFrameAsync and DeserializeFrameAsync. The frame boundary is transport metadata and does not alter the MemoryPack payload.

Generated formatters and the generic runtime path are designed for Native AOT. Reflection-based cold shape discovery must be preserved when trimming an application that relies on runtime generic shapes.

The type of T defined in Serialize<T> and Deserialize<T> is called C# schema. MemoryPack format is not self-described format. Deserialize requires the corresponding C# schema. These types exist as internal representations of binaries, but types cannot be determined without a C# schema.

Endian must be Little Endian. However, reference C# implementation does not care about endianness so can not use on big-endian machine. However, modern computers are usually little-endian.

There are eight types of format.

• Unmanaged struct
• Object
• Version Tolerant Object
• Circular Reference Object
• Tuple
• Collection
• String
• Union

Unmanaged struct is C# struct that doesn't contain reference types, similar constraint of C# Unmanaged types. Serializing struct layout as it is, includes padding.

(byte memberCount, [values...])

Object has 1byte unsigned byte as member count in header. Member count allows 0 to 249, 255 represents object is null. Values store memorypack value for the number of member count.

(byte memberCount, [varint byte-length-of-values...], [values...])

Version Tolerant Object is similar as Object but has byte length of values in header. varint follows these spec, first sbyte is value or typeCode and next X byte is value. 0 to 127 = unsigned byte value, -1 to -120 = signed byte value, -121 = byte, -122 = sbyte, -123 = ushort, -124 = short, -125 = uint, -126 = int, -127 = ulong, -128 = long.

(byte memberCount, [varint byte-length-of-values...], varint referenceId, [values...])
(250, varint referenceId)

Circular Reference Object is similar as Version Tolerant Object but if memberCount is 250, next varint(unsigned-int32) is referenceId. If not, after byte-length-of-values, varint referenceId is written.

(values...)

Tuple is fixed-size, non-nullable value collection. In .NET, KeyValuePair<TKey, TValue> and ValueTuple<T,...> are serialized as Tuple.

(int length, [values...])

Collection has 4 byte signed integer as data count in header, -1 represents null. Values store memorypack value for the number of length.

(int utf16-length, utf16-value)
(int ~utf8-byte-count, int utf16-length, utf8-bytes)

String has two-forms, UTF16 and UTF8. If first 4byte signed integer is -1, represents null. 0, represents empty. UTF16 is same as collection(serialize as ReadOnlySpan<char>, utf16-value's byte count is utf16-length * 2). If first signed integer <= -2, value is encoded by UTF8. utf8-byte-count is encoded in complement, ~utf8-byte-count to retrieve count of bytes. Next signed integer is utf16-length, it allows -1 that represents unknown length. utf8-bytes store bytes for the number of utf8-byte-count.

(byte tag, value)
(250, ushort tag, value)

First unsigned byte is tag that for discriminated value type or flag, 0 to 249 represents tag, 250 represents next unsigned short is tag, 255 represents union is null.

This library is licensed under the MIT License.