Skip to content

Cluster Mempool#6368

Draft
mononaut wants to merge 9 commits intomasterfrom
mononaut/cluster-mempool-backend
Draft

Cluster Mempool#6368
mononaut wants to merge 9 commits intomasterfrom
mononaut/cluster-mempool-backend

Conversation

@mononaut
Copy link
Copy Markdown
Contributor

@mononaut mononaut commented Mar 7, 2026

First draft of a Cluster Mempool implementation & integration with the main backend.

This PR adds a relatively self-contained typescript implementation of the Cluster Mempool algorithm under backend/src/cluster-mempool.

The module accepts a dictionary of transactions of the same form as the mempoolCache maintained by mempool.ts, optionally a dictionary of accelerations, constructs mempool clusters and chunkings, and saves cluster/chunk info and effective fee rates back to the transaction objects.

It then accepts mempool diffs (txs added, changed or removed), and incrementally applies these to efficiently update the clusters, chunkings and effective fee rates.


There are some vibecoded unit tests in backend/src/__tests__/cluster-mempool, as well as an end-to-end harness for comparing the templates generated by our cluster mempool module to those returned from getblocktemplate by a Bitcoin Core node running v30.99+

The e2e test harness seems to show that the generated templates are an exact match, but while the mempool is empty and everyone is using versions of Core without cluster mempool there aren't many exotic clusters in the mempool to exercise this thoroughly.

Moreover, if I understand correctly, Core's cluster mempool implementation isn't entirely deterministic in a way that we can reproduce, so even if the algorithm is correct we may see occasional mismatched templates in the e2e harness.


This cluster mempool module is integrated into the main backend behind a new config flag MEMPOOL.CLUSTER_MEMPOOL (default false).

When enabled, the backend uses the new cluster mempool module for calculating effective fee rates for txs in the mempool and for projecting blocks.


If audits are enabled, blocks above a certain activation height will be audited using both the cluster mempool algorithm and the legacy GBT algorithm (either typescript or rust versions, depending on config), and then we choose the audit with the best score (assuming that was the method actually used to construct the block). we detect which algorithm was likely used to construct the block*, and then conduct the audit against a projected block produced with the corresponding algorithm.

The choice of audit is then stored in the blocks-audits table.

*this heuristic works by calculating effective fee rates for the block using either cluster mempool or legacy GBT, and then running our prioritized transaction detection algorithm on the contents of the block using each set of fee rates. this efficiently identifies which set of fee rates is most parsimonious, and therefore which algorithm was likely used which high accuracy.


If CPFP indexing is enabled, new blocks that appeared to have been constructed using cluster mempool will be indexed using cluster mempool effective fee rates & cluster/chunk info.

We save that info to the existing compact_cpfp_clusters table by smuggling it into the existing txs column in a compact binary format, and mark them with a flag in a new template_algo column to identify which format is being used.


Leaving this in draft for now for initial review while I keep testing the main cluster mempool implementation for correctness, and check that the cpfp indexing & audit stuff works as intended.

@cla-bot cla-bot bot added the cla-signed label Mar 7, 2026
@wiz wiz requested a review from glozow March 9, 2026 13:00
@mononaut mononaut force-pushed the mononaut/cluster-mempool-backend branch from 964a563 to 9aa0ec7 Compare March 10, 2026 06:56
@mononaut mononaut force-pushed the mononaut/cluster-mempool-backend branch from 6da80e6 to 38984ae Compare March 20, 2026 05:45
@mononaut
Copy link
Copy Markdown
Contributor Author

  • rebased on master
  • considerable refactor of the new block pipeline, in order to:
    • fix some issues with audits & cpfp indexing
    • significantly improve performance, esp. during cpfp indexing

instead of conducting dual audits & deciding based on the audit score, we now detect which algorithm was likely used to construct a block using a new metric based around apparent prioritizations which is much more accurate & discriminating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants