Darp.BinaryObjects

A source generator to generate TryRead/Write Little/BigEndian methods for struct/class definitions.

Important

This package is under heavy development. Anything is subject to change.

You should use the source generation when you want:

• Serialization to a buffer of bytes
• Deserialization from a buffer already completely received
• Endianness during serialization
• Common interfaces for serialization are required which allow implementation of more complex scenarios by hand without the generator
• Usage of something like BinaryPrimitives but for more complex types
• Can work with a minimum c# LanguageVersion of 11 and net9.0 / net10.0

If these requirements do not meet your expectations, check out those other wonderful projects

• Several binary serializers. e.g. MemoryPack, BinaryPack, ... which are great if direct binary serialization is not needed
• Serialization libraries relying on reflection. e.g. HyperSerializer
• StructPacker - not supporting allocation less packing/unpacking
• BinarySerializer - Allows for binary serialization with a way larger feature set but more difficult to understand and relying on reflection

Supported properties

Here is a list of the property types currently supported by the library:

• Unmanaged types: bool, sbyte, byte, short, ushort, int, uint, long, ulong, char, float, double
• BinaryObjects implementing IBinaryWritable or IBinaryReadable
• Blittable types
• Enums
• Other .NET types: BitArray

For all of these types, it should be possible to define as array types:

• Memory abstractions: ReadOnlyMemory<T>
• Arrays: T[]
• Lists: List<T>
• Counted collections: IReadOnlyCollection<T>, ICollection<T>, IReadOnlyList<T>, IList<T>

IEnumerable<T> members are not supported, including with BinaryElementCount. Materialize lazy sequences with ToArray() or ToList() and declare the member as an array, list, or supported counted collection interface. Sizing uses Length or Count without enumerating the collection. Keep collections unchanged between sizing and writing, and during serialization.

To control these types there are attributes

• BinaryIgnore: Ignore some members
• BinaryElementCount: Sets the number of elements in an array
• BinaryReadRemaining: Reads the remaining into an array
• BinaryByteCount: Sets the number of bytes of an integer or enum which is narrower than its type, or of a collection
• BinaryElementByteCount: Sets the number of bytes of each integer or enum in a collection
• BinaryConstantValue: Mark a (readonly) property which will have a predefined value

Unplanned:

• Unmanaged types have no clearly defined length / endianness: nint, nuint, decimal
• Multidimensional arrays (e.g. T[,], T[,,], etc.)
• Jagged arrays (e.g. T[][], etc.)
• Dictionaries: Dictionary<TKey, TValue>, IDictionary<TKey, TValue> and IReadOnlyDictionary<TKey, TValue>
• Nullable value types: Nullable<T> or T?

What is serialized?

• Any real, user-defined member in a class or struct declaration

• Any field or auto property which is settable or has a parameter with matching type and name in the constructor

• The instance constructor marked with BinaryConstructorAttribute is used, regardless of declaration order or accessibility. Readers call it; readers and writers match its parameters to members.

• Without that attribute, the sole explicit instance constructor is used; types without one use their implicit parameterless constructor.

• Static constructors and compiler-generated constructors (including record copy constructors) do not participate in selection.

• Multiple explicit instance constructors require exactly one BinaryConstructorAttribute; otherwise generation fails with DBO010.

There are warnings if:

• A member is readonly and does not have a matching constructor argument or is explicitly ignored

Choosing generated methods

[BinaryObject] defaults to BinaryOptions.All, generating IBinaryObject<T> with both readers and writers. Choose a direction when only one is needed:

[BinaryObject(BinaryOptions.Read)]
public sealed partial record Incoming(ushort Value, byte[] Data);

[BinaryObject(BinaryOptions.Write)]
public sealed partial record Outgoing(ushort Value, byte[] Data);

Read generates IBinaryReadable<T> and TryReadLittleEndian / TryReadBigEndian. Write generates IBinaryWritable, GetByteCount, and TryWriteLittleEndian / TryWriteBigEndian. Both directions include overloads that report the consumed or written byte count. Methods in the omitted direction can be implemented by hand. Nested objects must support the direction their parent uses. For any manual serializer, including IBinaryObject<T>, use BinaryConstant when a fixed byte length is known; otherwise nested scalar objects use the byte counts reported by their implementation. Only members declared on an annotated type are serialized. An annotated class with a base class triggers warning DBO005; suppress it when the base class is intentional. The generator does not infer a handwritten serializer's wire layout from its fields or properties. Object collections require a positive fixed binary element length. BinaryElementCount defines how many elements are present; manual element types also need BinaryConstant to define each element's size.

Collections without BinaryElementCount consume all complete elements remaining in the supplied input span. They must be the last serialized member, including on write-only objects; ignored and computed members do not affect this rule. Pass a span bounded to one message when reading such objects. A nested object that consumes the remaining input also needs a bounded span if its parent has trailing data. Use a constant or member-defined BinaryElementCount, or a member-defined BinaryByteCount, for collections followed by other serialized members.

Generated readers and writers return false for negative counts or counts that cannot fit the supplied buffer. Writers also return false when a collection has fewer elements than its declared count or minimum count. A declared BinaryElementCount writes only that many elements; surplus elements are ignored. Constant counts and minimum lengths that cannot form a valid int byte length are rejected during generation. GetByteCount throws OverflowException when the required byte count exceeds int.MaxValue. Failed reads and writes may report partial progress; a failed write can modify the destination. Nested child serializers returning false cause the generated parent to return false, including for fixed-size objects and object collections. The parent includes the child's reported progress on failure. Exceptions thrown by the child itself propagate. Fixed-size children and collection elements retain their declared slices and strides on success, even when a manual serializer reports fewer bytes.

Write-only objects can serialize readonly fields and getter-only auto properties without matching constructor parameters; they do not need to be reconstructible by the generated reader. [BinaryObject((BinaryOptions)0)] disables generation, including member diagnostics.

Integers narrower than their type

BinaryByteCount reads and writes an integer or enum with fewer bytes than its type has, such as a 24-bit value in a uint. BinaryElementByteCount does the same for each element of a collection:

[BinaryObject]
public sealed partial record Samples(
    [property: BinaryByteCount(3)] uint Timestamp,
    [property: BinaryByteCount(3)] int Count,
    [property: BinaryElementCount("Count"), BinaryElementByteCount(3)] int[] Values
);

• Both apply to sbyte, byte, short, ushort, int, uint, long, ulong and enums. Other types fail with DBO013.
• A constant BinaryByteCount on a collection fails with DBO014 and BinaryElementByteCount on a single value with DBO015.
• The byte count has to be between 1 and the size of the type; other values fail with DBO012. A byte count equal to the size of the type has no effect and reports the informational DBO011.
• Readers zero-extend unsigned types and sign-extend signed types. Enums follow their underlying type.
• Writers return false when a value does not fit into its byte count. A scalar that does not fit fails before any of its adjacent fixed-size members are written; a collection stops at the first element that does not fit.
• A member referenced by BinaryElementCount still has to be an sbyte, byte, short, ushort or int.

Collections bounded by a byte count

BinaryByteCount with a member name ends a collection after the number of bytes the member holds, where BinaryElementCount counts elements:

[BinaryObject]
public sealed partial record Packet(
    ushort PayloadLength,
    [property: BinaryByteCount("PayloadLength")] int[] Values,
    byte Checksum
);

A PayloadLength of 8 reads and writes two int values.

• The elements need a fixed size: primitives, enums, elements narrowed with BinaryElementByteCount, and objects with a constant binary length.
• The member has to be serialized before the collection and be an sbyte, byte, short, ushort or int, as for BinaryElementCount.
• Readers and writers return false when the byte count is negative, exceeds the buffer or is not a multiple of the element size.
• Writers write as many elements as the byte count holds, return false when the collection has fewer, and ignore surplus elements. GetByteCount uses the byte count without validating it.
• BinaryMinElementCount applies to the number of elements the byte count holds.
• Combining it with BinaryElementCount fails with DBO016; a member name on a single value fails with DBO017.

How it's supposed to work

Let's pretend we have a series of bytes:

01020003040506

A: 01
B: 0200
Data: 03040506

Normally, you would have to write serialization methods for yourself. By adding the BinaryObjectAttribute, this is done automatically by the source generator.

This:

public readonly record struct SomeTestStruct(byte A, ushort B, ReadOnlyMemory<int> Data);

bool TryReadSomeTestStruct(ReadOnlySpan<byte> source, out SomeTestStruct value)
{
    if (source.Length < 3)
    {
        value = default;
        return false;
    }
    var a = source[0];
    var b = BinaryPrimitives.ReadUInt16LittleEndian(source[1..]);
    var dataArray = new int[(source.Length - 3) / sizeof(int)];
    ReadOnlySpan<int> reinterpretedData = MemoryMarshal.Cast<byte, int>(source[2..]);
    if (BitConverter.IsLittleEndian)
    {
        reinterpretedData.CopyTo(dataArray);
    }
    else
    {
        BinaryPrimitives.ReverseEndianness(reinterpretedData, dataArray);
    }
    value = new SomeTestStruct(a, b, dataArray);
    return true;
}

TryReadSomeTestStruct(buffer, out SomeTestStruct value);

Becomes this:

[BinaryObject]
public readonly partial record struct SomeTestStruct(byte A, ushort B, ReadOnlyMemory<int> Data);

SomeTestStruct.TryReadLittleEndian(buffer, out SomeTestStruct value);

Usage

// Define your object
[BinaryObject]
partial record struct YourStruct(ushort A, byte B);

// Read the struct from the buffer using either little or big endian format
var buffer = Convert.FromHexString("AABBCC");
var success = YourStruct.TryReadLittleEndian(source: buffer, out var value);
var success2 = YourStruct.TryReadBigEndian(source: buffer, out var value2, out int bytesRead);

// Get the actual size of the struct
var size = value.GetByteCount();

// Write the values back to a buffer
var writeBuffer = new byte[size];
var success3 = value.TryWriteLittleEndian(destination: writeBuffer);
var success4 = value2.TryWriteLittleEndian(destination: writeBuffer, out int bytesWritten);

The code generated by the struct will attempt to maximize readability by still maintaining performance and as little allocations as possible.

Generated code

// <auto-generated/>
#nullable enable

using BinaryHelpers = global::Darp.BinaryObjects.BinaryHelpers;
using NotNullWhenAttribute = global::System.Diagnostics.CodeAnalysis.NotNullWhenAttribute;

namespace Your.Namespace;

/// <remarks> <list type="table">
/// <item> <term><b>Field</b></term> <description><b>Byte Length</b></description> </item>
/// <item> <term><see cref="A"/></term> <description>2</description> </item>
/// <item> <term><see cref="B"/></term> <description>1</description> </item>
/// <item> <term> --- </term> <description>3</description> </item>
/// </list> </remarks>
public partial record struct YourStruct : global::Darp.BinaryObjects.IBinaryWritable, global::Darp.BinaryObjects.IBinaryReadable<YourStruct>
{
    /// <inheritdoc />
    [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
    public int GetByteCount() => 3;

    /// <inheritdoc />
    public bool TryWriteLittleEndian(global::System.Span<byte> destination) => TryWriteLittleEndian(destination, out _);
    /// <inheritdoc />
    public bool TryWriteLittleEndian(global::System.Span<byte> destination, out int bytesWritten)
    {
        bytesWritten = 0;

        if (destination.Length < 3)
            return false;
        BinaryHelpers.WriteUInt16LittleEndian(destination[0..], this.A);
        BinaryHelpers.WriteUInt8(destination[2..], this.B);
        bytesWritten += 3;

        return true;
    }
    /// <inheritdoc />
    public bool TryWriteBigEndian(global::System.Span<byte> destination) => TryWriteBigEndian(destination, out _);
    /// <inheritdoc />
    public bool TryWriteBigEndian(global::System.Span<byte> destination, out int bytesWritten)
    {
        bytesWritten = 0;

        if (destination.Length < 3)
            return false;
        BinaryHelpers.WriteUInt16BigEndian(destination[0..], this.A);
        BinaryHelpers.WriteUInt8(destination[2..], this.B);
        bytesWritten += 3;

        return true;
    }

    /// <inheritdoc />
    public static bool TryReadLittleEndian(global::System.ReadOnlySpan<byte> source, out YourStruct value) => TryReadLittleEndian(source, out value, out _);
    /// <inheritdoc />
    public static bool TryReadLittleEndian(global::System.ReadOnlySpan<byte> source, out YourStruct value, out int bytesRead)
    {
        bytesRead = 0;
        value = default;

        if (source.Length < 3)
            return false;
        var ___readA = BinaryHelpers.ReadUInt16LittleEndian(source[0..]);
        var ___readB = BinaryHelpers.ReadUInt8(source[2..]);
        bytesRead += 3;

        value = new YourStruct(___readA, ___readB);
        return true;
    }
    /// <inheritdoc />
    public static bool TryReadBigEndian(global::System.ReadOnlySpan<byte> source, out YourStruct value) => TryReadBigEndian(source, out value, out _);
    /// <inheritdoc />
    public static bool TryReadBigEndian(global::System.ReadOnlySpan<byte> source, out YourStruct value, out int bytesRead)
    {
        bytesRead = 0;
        value = default;

        if (source.Length < 3)
            return false;
        var ___readA = BinaryHelpers.ReadUInt16BigEndian(source[0..]);
        var ___readB = BinaryHelpers.ReadUInt8(source[2..]);
        bytesRead += 3;

        value = new YourStruct(___readA, ___readB);
        return true;
    }
}

Development

Open Darp.BinaryObjects.slnx to work with the full solution.

After cloning the repository, you will find the following project structure:

• src/Darp.BinaryObjects contains public APIs and Attributes
• src/Darp.BinaryObjects.Generator contains the actual source generator
• test/Darp.BInaryObjects.Generator.Tests contains snapshot tests verifying the files generated by the source generator
• test/Darp.BinaryObjects.Tests contains unit tests ensuring the generated files actually valid

Code formatting

This repository uses CSharpier (inspired by prettier) for code formatting. Install the local tool explicitly with dotnet tool restore.

To run it, execute

dotnet csharpier format .

If you want to format you code on save, check out available Editor integration for your IDE.

Testing

Development requires the .NET 10 SDK selected by global.json and the .NET 9 runtime. Both test projects use xUnit v3 with Microsoft.Testing.Platform and run on .NET 9 and .NET 10:

dotnet test

Collect coverage with dotnet test --coverlet.

Package versions are managed centrally in Directory.Packages.props.

Snapshot tests are done using Verify. If you want to optimize running these tests in your local IDE, you might adjust some settings. Please, check your local configuration in the VerifyDocs

Releases

Conventional commits on main are collected by release-please into a release PR. Merging that PR updates the shared package version and changelog, creates a GitHub release, and builds, tests, and publishes the package to NuGet using the NUGET_API_KEY repository secret. Release PR checks created with GITHUB_TOKEN may require a maintainer to approve their workflow runs.