Skip to content

[NO-REVIEW] Use Zstandard for compressing singlefile assemblies#123542

Draft
rzikm wants to merge 2 commits intodotnet:mainfrom
rzikm:zstd-bundling
Draft

[NO-REVIEW] Use Zstandard for compressing singlefile assemblies#123542
rzikm wants to merge 2 commits intodotnet:mainfrom
rzikm:zstd-bundling

Conversation

@rzikm
Copy link
Member

@rzikm rzikm commented Jan 23, 2026

This is an experiment that replaces DEFLATE with Zstandard in single-file published applications for compression of bundled managed assemblies.

Edit: This has been quite complicated to figure out how to test, documenting the steps below for future reference.

Resulting csproj file:
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <RootNamespace>test_app</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <!-- Compression is opt-in -->
    <EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>

    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <PublishSingleFile>true</PublishSingleFile>
    <SingleFileHostSourcePath>C:\source\dotnet\runtime\artifacts\bin\win-x64.Release\corehost\singlefilehost.exe</SingleFileHostSourcePath>
  </PropertyGroup>

<ItemGroup>
  <FrameworkReference Update="Microsoft.NETCore.App" RuntimeFrameworkVersion="11.0.0-dev" />
</ItemGroup>

</Project>
NuGet.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <!-- use local nuget cache which you can delete between builds -->
    <add key="globalPackagesFolder" value="C:\source\repros\2026-02-02-zstd-bundling\test-app\nuget_cache" />
  </config>

  <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />
    <!-- <add key="nuget" value="https://api.nuget.org/v3/index.json" /> -->
    <add key="dotnet11" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11/nuget/v3/index.json" />
    <add key="local runtime" value="D:\dotnet\runtime\artifacts\packages\Release\Shipping" />
  </packageSources>
</configuration>
  • If changes to the singlefilehost are made (e.g. peimagelayout.cpp), entire Clr subset
  • If you make changes to the bundler (Microsoft.NET.HostModel.dll) delete the bin/obj folders of the test application to force repacking.

The testing application was a simple ASP.NET hello world with the weather API, with immediate exit after startup

if (args.Length == 0)
{
    return;
}

Startup speed impact

I temporarily put extra logs in the apphost to verify that we decompress at least some images, and there seem to have been about 20 of them. Startup times themselves had quite a multimodal distribution (probably due to filesystem caching), the times below show the lowest modality:

hyperfine --warmup 10 --runs 10 Min Max Mean Std dev
Compression disabled 69.5 ms 94.8 ms 79.9ms 7.0 ms
Deflate 119.8 ms 124.9 ms 122.0 ms 1.4 ms
Zstd (SmallestSize) 118.0 ms 121.8 ms 120.4 ms 1.2 ms

I would expect similar diffs on higher modalities, there was no difference across different compression level options that I could measure, so I did not include more lines in the table above

Build time and binary size impact:

For measuring build times and actual binary sizes I measured the execution of dotnet publish ... after deleting bin and obj folders (keeping the nuget_cache intact).

Method Mean Std dev Min Max Binary size Size diff
Compression disabled 3.740 s 0.106s 3.600 s 3.886 s 102,889,121 B 106.90%
Zstandard (Quality=1) 4.515 s 0.123 s 4.418 s 4.671 s 54,271,488 B 9.14%
Zstandard (Quality=2) 4.553 s 0.200 s 4.293 s 4.848 s 52,390,638 B 5.35%
Zstandard (Quality=3) 4.746 s 0.089 s 4.645 s 4.880 s 50,995,793 B 2.55%
Zstandard (Quality=4) 4.998 s 0.271 s 4.698 s 5.410 s 50,819,706 B 2.2%
Zstandard (Quality=5) 5.196 s 0.283 s 4.837 s 5.563 s 50,122,524 B 0.79%
Deflate 6.888 s 0.087 s 6.776 s 7.035 s 49,728,027 B -
Zstandard (Quality=6) 5.348 s 0.118 s 5.186 s 5.451 s 49,516,508 B -0.43%
Zstandard (Quality=7) 5.788 s 0.326 s 5.453 s 6.307 s 49,299,101 B -0.86%
Zstandard (Quality=8) 5.862 s 0.105 s 5.681 s 5.946 s 49,209,491 B -1.04%
Zstandard (Quality=9) 6.282 s 0.280 s 5.837 s 6.583 s 49,184,329 B -1.09%
Zstandard (Quality=10) 6.614 s 0.194 s 6.287 s 6.808 s 49,121,708 B -1.22%
Zstandard (Quality=11) 6.797 s 0.320 s 6.397 s 7.201 s 49,086,309 B -1.29%
Zstandard (Quality=12) 7.071 s 0.359 s 6.812 s 7.673 s 49,081,407 B -1.3%
Zstandard (Quality=13) 10.724 s 4.174 s 8.620 s 18.187 s 48,843,435 B -1.78%
Zstandard (Quality=14) 9.556 s 0.292 s 9.128 s 9.808 s 48,512,643 B -2.44%
Zstandard (Quality=15) 9.842 s 0.219 s 9.562 s 10.161 s 48,498,631 B -2.47%
Zstandard (Quality=16) 14.723 s 0.795 s 14.228 s 16.137 s 47,187,005 B -5.11%
Zstandard (Quality=17) 14.984 s 0.126 s 14.870 s 15.179 s 46,609,240 B -6.27%
Zstandard (Quality=18) 19.173 s 0.246 s 18.945 s 19.542 s 45,003,552 B -9.5%
Zstandard (Quality=19) 23.365 s 1.071 s 22.347 s 24.968 s 44,972,958 B -9.56%
Zstandard (Quality=20) 23.067 s 0.679 s 22.050 s 23.826 s 44,964,647 B -9.58%
Zstandard (Quality=21) 21.588 s 1.638 s 19.780 s 23.580 s 44,964,082 B -9.58%
Zstandard (Quality=22) 24.276 s 0.587 s 23.616 s 25.139 s 44,963,834 B -9.58%

Looks like for Quality values between 5 and 11, we start getting small gains in both startup time and binary size, but they seem rather small, we can squeeze some more gains by training a dictionary on the usual dll files (runtime+aspnet?), but this shaves only about additional 1MB at most, so is likely not worth the extra logistics around embedding the decompression dictionary in the singlefile exe

@github-actions github-actions bot added the area-HostModel Microsoft.NET.HostModel issues label Jan 23, 2026
@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @agocke, @elinor-fung
See info in area-owners.md if you want to be subscribed.

@rzikm
Copy link
Member Author

rzikm commented Jan 23, 2026

@VSadov, @vitek-karas, @janvorli do you know what would be the easiest way to test/benchmark this kind of change?

@rzikm rzikm changed the title [DRAFT] Use Zstandard for compressing singlefile assemblies [NO-REVIEW] Use Zstandard for compressing singlefile assemblies Jan 23, 2026
@rzikm rzikm added the NO-REVIEW Experimental/testing PR, do NOT review it label Jan 23, 2026
@am11
Copy link
Member

am11 commented Jan 23, 2026

@VSadov, @vitek-karas, @janvorli do you know what would be the easiest way to test/benchmark this kind of change?

./build.sh clr+libs+packs -c Release, extract artifacts/packages/Shipping/xxx/dotnet-runtime-xxx.tar.gz into ./.dotnet2 and copy sdk dir ./.dotnet/sdk to ./dotnet2/sdk, adjust the versioned folder to match the shared framework (either change sfx version to match sdk or the other way around; I don't remember exactly), then cd .. && runtime/.dotnet2/dotnet new console -n helloworld && cd helloworld && ../runtime/.dotnet2/dotnet publish -p:PublishSingleFile=true -o dist && dist/helloworld. If you see Hello World!, you pass! :)

@jkotas
Copy link
Member

jkotas commented Jan 23, 2026

dotnet new console

For benchmarking, you may want to pick an app with more managed code, like self-contained ASP.NET webapi. We want to measure the binary and startup time (just make the app exit immediately and then you can just measure the time the process took to run).

Copilot AI review requested due to automatic review settings February 6, 2026 07:42
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This is an experimental pull request that replaces DEFLATE compression with Zstandard compression for bundled managed assemblies in single-file published applications. The PR is marked as "[NO-REVIEW]" indicating it's a prototype or work-in-progress.

Changes:

  • Native decompression code updated from zlib to Zstandard in both bundle extraction and PE image layout modules
  • Managed compression code updated to use ZstandardStream (for .NET) instead of DeflateStream
  • Build system updated to reference Zstandard headers instead of zlib headers

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/native/corehost/bundle/extractor.h Added zstd.h include and ZSTD_DCtx member for reusable decompression context with proper cleanup in destructor
src/native/corehost/bundle/extractor.cpp Replaced zlib decompression logic with Zstandard streaming decompression, removed pal_zlib.h include
src/installer/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj Added reference to System.IO.Compression.Zstandard for .NET builds
src/installer/managed/Microsoft.NET.HostModel/Bundle/Bundler.cs Updated compression logic to use ZstandardStream with SmallestSize compression level for .NET builds, keeping DeflateStream for .NET Framework
src/coreclr/vm/peimagelayout.cpp Replaced zlib decompression with Zstandard single-shot decompression for compressed PE images
src/coreclr/vm/CMakeLists.txt Updated include directory from System.IO.Compression.Native to external/zstd/lib

@rzikm
Copy link
Member Author

rzikm commented Feb 6, 2026

I finally managed to measure some numbers, please take a look a the data in the top post. Is the difference enough for us to consider taking the change? we can also later explore lzma compression once it gets implemented later in the release cycle.

@jkotas
Copy link
Member

jkotas commented Feb 6, 2026

I think the interesting part is the tail end Zstandard (Quality=22) that allows you to burn extra build time to get smaller binaries. Does Deflate come with similar quality config - are the numbers for highest Deflate quality comparable to Zstandard?

@rzikm
Copy link
Member Author

rzikm commented Feb 6, 2026

Does Deflate come with similar quality config

The bundler is using Deflate with CompressionLevel.SmallestSize, so I think it already is using the highest setting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-HostModel Microsoft.NET.HostModel issues NO-REVIEW Experimental/testing PR, do NOT review it

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants