Transaction Pools

A transaction pool or mempool is a data structure containing the set of transactions that have not been mined but have been validated by a transaction processor (or miner). A node has different kinds of transaction pools, and depending on how a transaction has been classified it may be stored in one of them. An expectation exists that if a transaction is submitted to a transaction processing service, or broadcast on the P2P network, it will result in that transaction being known and held in a pool on each of the mining nodes throughout the network.

Types of transaction pool

There is only one pool of each type, and each serves a different purpose.

The Mempool

This is the set of validated complete transactions that are known to the miner and meet at least minimum relay fee. It includes both the transactions in the mempool proper that meet the criteria (including the minimum transaction fee) to be included in any new block they may create, and the transactions in the secondary mempool that do not meet that criteria.

Transaction Addition

Acceptance Any new transaction that arrives cannot of course conflict with any existing mempool transaction. Additionally, given that Bitcoin SV script engineers can write flexible and extensive scripts that can do any number of interesting things, miners need to impose some limit on how long they spend validating the more complicated of these scripts. At this time, if a script takes longer than a second to validate, it is dropped. The mempool is of course also limited in size, and once it becomes full restrictions are imposed until it is sufficiently empty. A minimum fee rate is imposed, and transactions must pay over this fee rate to be accepted into the mempool.

The Secondary Mempool The transactions that do not meet one miner’s criteria to be included in a block, but may meet others, are kept and considered to be the secondary mempool. These transactions will typically have below the minimum transaction fee limit but above the network-wide minimum relay fee limit. In the past these transactions were dropped but given that different nodes may have different configurations, this results in each potentially filtering out and ignoring transactions that do not meet their criteria, and having different transaction pools. The primary benefit of the secondary mempool is that the transactions contained within it could eventually be included in another transaction processor's block and retaining them ensures fast block validation and optimizes network performance. Without the secondary mempool if a user can identify the disparity in node configurations, they know they can submit different transactions spending the same coins to different miners. This can be used to double spend an earlier transaction.

Transaction Removal

Block Creation When creating a new block to mine, miners will gather transactions from this pool using the rpc command getminingcandidate or the older getblocktemplate to construct a candidate block. Similarly, when receiving a block, a validator can speed-up the validation process if the transactions IDs match the transaction IDs in the validator's transaction pool.

Pool Expiry Transactions contained in this pool are checked for expiry any time a new transaction is added to it. Transactions older than the expiry age are first removed, and then additional transactions are removed to bring the size below the maximum allowed size working from lowest fee transaction to highest fee transaction. The process of removing transactions in this way bumps the minimum fee rate for acceptance.

Current default settings:

• The maximum size of this pool is currently set to 1 GB [6]
• The expiry age is currently set to 14 days (336 hours) [7]

Child Pays for Parent

In the event that a user managed to get their transaction in the mempool, where the fee was high enough that the transaction was a candidate for mining but not high enough that it would get mined promptly, they have the option of using “child pays for parent.” The way this works is that a change output from the low fee transaction is used to make a child transaction with a high enough fee that it pays for both the parent and the child. Now that even transactions a miner would not consider mining can end up being stuck in the secondary mempool, this becomes even more useful to promote them into the mempool proper.

The Non-Final Pool

The protocol allows for applications to broadcast transactions that may still undergo further revisions from the parties involved, and the non-final pool is used to track these transactions. When they become finalised either because the participants mark them as such, or a declared finalisation time marked by either block height or timestamp is passed, they are moved to the primary transaction pool. Applications submitting non-final transactions should be aware they are committing to a payment, and if a submitted transaction will finalize before it would expire out of the pool, the payment would go ahead. If this is not pending and not desirable, they will have to negotiate with the other participants to overwrite a submitted transaction.

Transaction Addition

The decision whether to put a transaction in the final or non-final mempool is done after all other validation checks have passed. This means that the criteria for entering beyond a transaction qualifying as non-final, are the same as entering the main mempool. The reference for what constitutes a non-final transaction can be found in the Genesis upgrade specification [8]. However there are additional limitations to which transactions can be added into this pool, one example of this is if a non-final transaction is being considered for addition and it spends the outputs of other non-final transactions, it will be rejected [2]. Even if the node considers the transaction valid and might add it to the non-final pool, it also will reject the transaction if there is no remaining space in the pool [1].

Current default settings:

• The default maximum size of this pool is 50 MiB [5]

Transaction Removal

The non-final pool performs periodic checks on all of the transactions within it, and this may either detect that one of the previously non-final transactions has become finalized or should be dropped due to lack of activity.

Finalisation If the lock time of a transaction has passed, this overrides all of the other conditions that make a transaction non-final, and it becomes a candidate for movement to the primary pool.

Pool Expiry Any transaction that has remained in this pool without being updated for longer than the maximum entry age, is dropped [4].

Current default settings:

• The default expiry age is 4 weeks [3]

Evolution of the Mempool

With the addition of the secondary mempool concept and the restoration of the non-final mempool, Bitcoin SV has come a long way since the first release of Bitcoin. This section outlines the changes that have been made.

The Mempool in the original Satoshi client

In the original source code, the “mempool” was in fact a temporary in-memory pool of transactions. If the node was restarted, then the node would have to repopulate it’s mempool from other nodes. If all nodes shut down at the same time, then the mempool would be empty and anyone wanting their transaction mined would need to rebroadcast.

The notion of non-final transactions was part of the original design:

1. If the transaction lock time was 0, it was final.
2. Otherwise, if the transaction lock time was less than the height of the longest chain, it was final.
3. Otherwise, if any transaction input had a sequence number lower than the highest possible value (0xFFFFFFFF) the transaction was non-final.
4. Otherwise, the transaction was final.

Transactions in the mempool were retained as long as the node continued running. The mining code considered them and ruled them out based on whether they were final or not. Those that had a pending lock time would eventually become final, and viable for mining.

Transactions with too low a fee had a way to get mined, as the first 100 transactions in each block given that they were under 10 kilobytes in size were free.

The Bitcoin Core mempool

At some point along the way, persistence was added to the mempool and an expiry period for transactions that remained in there past a certain amount of time. Non-final transactions became undesirable, and were no longer accepted. The mempool was now a place where only immediately mineable transactions were found.

Transactions in the mempool expire after a default period of 2 weeks, although miners can override this. Anyone whose transaction was stuck in the mempool, and wasn’t getting mined because the fee was too low, could now do something called child pays for parent. By paying a higher fee for a transaction whose parent was still unconfirmed, it’s fee would be counted towards the fee of it’s parent for the purposes of getting that parent mined in a timely fashion.

The lock time field of a transaction was also repurposed by Bitcoin Core, with a new magic value of 500,000,000. If the value is above this, it is interpreted as a timestamp that is compared to a block time according to BIP 113. Values below this continue to be interpreted as a block height.

Sequence numbers of transaction inputs were similarly repurposed by Bitcoin Core according to BIP 68. These now had timestamp-based constraints on when they could be mined.

References

[1] mempool inclusion size limit

[2] mempool tx spending another non-final tx disallowed

[3] time a tx can stay in the non-final mempool

[4] non-final pool check drops expired tx

[5] pool default maximum size

[6] mempool default maximum size

[7] mempool default expiry age (hours)

[8] Generic update specification non-final criteria section