Draft
Conversation
964a563 to
9aa0ec7
Compare
6da80e6 to
38984ae
Compare
Contributor
Author
instead of conducting dual audits & deciding based on the audit score, we now detect which algorithm was likely used to construct a block using a new metric based around apparent prioritizations which is much more accurate & discriminating. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First draft of a Cluster Mempool implementation & integration with the main backend.
This PR adds a relatively self-contained typescript implementation of the Cluster Mempool algorithm under
backend/src/cluster-mempool.The module accepts a dictionary of transactions of the same form as the
mempoolCachemaintained bymempool.ts, optionally a dictionary of accelerations, constructs mempool clusters and chunkings, and saves cluster/chunk info and effective fee rates back to the transaction objects.It then accepts mempool diffs (txs added, changed or removed), and incrementally applies these to efficiently update the clusters, chunkings and effective fee rates.
There are some vibecoded unit tests in
backend/src/__tests__/cluster-mempool, as well as an end-to-end harness for comparing the templates generated by our cluster mempool module to those returned from getblocktemplate by a Bitcoin Core node running v30.99+The e2e test harness seems to show that the generated templates are an exact match, but while the mempool is empty and everyone is using versions of Core without cluster mempool there aren't many exotic clusters in the mempool to exercise this thoroughly.
Moreover, if I understand correctly, Core's cluster mempool implementation isn't entirely deterministic in a way that we can reproduce, so even if the algorithm is correct we may see occasional mismatched templates in the e2e harness.
This cluster mempool module is integrated into the main backend behind a new config flag
MEMPOOL.CLUSTER_MEMPOOL(default false).When enabled, the backend uses the new cluster mempool module for calculating effective fee rates for txs in the mempool and for projecting blocks.
If audits are enabled,
blocks above a certain activation height will be audited using both the cluster mempool algorithm and the legacy GBT algorithm (either typescript or rust versions, depending on config), and then we choose the audit with the best score (assuming that was the method actually used to construct the block).we detect which algorithm was likely used to construct the block*, and then conduct the audit against a projected block produced with the corresponding algorithm.The choice of audit is then stored in the blocks-audits table.
*this heuristic works by calculating effective fee rates for the block using either cluster mempool or legacy GBT, and then running our prioritized transaction detection algorithm on the contents of the block using each set of fee rates. this efficiently identifies which set of fee rates is most parsimonious, and therefore which algorithm was likely used which high accuracy.
If CPFP indexing is enabled, new blocks that appeared to have been constructed using cluster mempool will be indexed using cluster mempool effective fee rates & cluster/chunk info.
We save that info to the existing
compact_cpfp_clusterstable by smuggling it into the existingtxscolumn in a compact binary format, and mark them with a flag in a newtemplate_algocolumn to identify which format is being used.Leaving this in draft for now for initial review while I keep testing the main cluster mempool implementation for correctness, and check that the cpfp indexing & audit stuff works as intended.