The Great Aggregating [v0.3.76rc1]

[indolering]

An entity (pool or exchange) is aggregating lots of low value outputs. As this is good for the system in regard to future pruning efforts it is currently causing client time outs for miners, thus they go offline a lot. The entity seems to own a lot of NMC so it probably is not a deliberate attack.

The transactions come in waves of a couple hundreds to thousands, they are 18kb (100 inputs) or 27kb (150 inputs) large. They only cost the relay fee / kb to relay. The functions underlying the RPC call "getauxblock" can take quite a long time to go through processing these.

What we are doing in the new version to fix the issue is caching input information necessary for the mining relevant RPC calls (thanks again Domob!). Also we will limit tx size to 20kb as it's worse with larger tx O(numberInputs * txSize) and increase the relay fee to 1/5 of the minTxFee.

Edit rc1: While the previous version had things up and working already, the changes contained in this version removed the last bottleneck and the system should now be able to properly handle even the large waves of aggregator txs.

Changelog

Everybody give it a shot:
https://github.com/namecoin/namecoin/tree/v0.3.76rc1
http://namecoin.info/files/Namecoin_v0. ... _setup.exe
sha256:

https://github.com/namecoin/namecoin/bl ... angelog.md

Relevant Github links:

• Add "mild" caching for priority calculation (latest in addition to a series of accepted smaller patches).
• Add more aggressive caching of previous txouts in CTxIn (closed but includes discussion of problem and fixes).
• getauxblock can take a long time

[text above edited by phelix as I could not get hold of Indolering]

~~~

Recently, someone tried to aggregate lots of Namecoin. Their intentions were positive (it reduces the UTXO set) but they did not pay the minimum fee on their transactions.

The transactions should not have propagated, however, MIN_RELAY_TX_FEE was set to MIN_TX_FEE/50. We released an emergency fix which upped the minimum relay fee.

However, sending a large number transactions with a large enough fee to be relayed but not rapidly mined likely affects other currencies, especially merged minded alts.

A permanent fix requires improving the performance of getmemorypool and getauxblock when mempool has a large number of transactions.

Please discuss.

[indolering]

Suggested edits for the OP are warmly welcomed.

I'm not a mainline dev, I'm primarily helping to coordinate the response.

[midnightmagic]

https://gist.github.com/midnightmagic/5 ... b3f738a56e

I have built a replacement statically-linked namecoind from commitid db10ddb1a712e499068b477af027a3110b3aad3b, 64-bit, from a Ubuntu 12.04.5 host.

For those of you out there who are running pools or exchanges, please *build* your own from the above commit, from:

https://github.com/namecoin/namecoin/tree/relay_tmp_fix

For those users who just want to help block the relay of these problematic transactions to buy us all time, please feel free to use the above binary.

You can tack on .to to the .onion (so, it becomes onion.to) but if you do, please verify the hash.

[indolering]

I'm just going to ask a dumb question here: The 'evil' tx have fees of 0.005 nmc attached to them right? This is std default client fee (also in main.h) and we want to raise the min_relay_tx_fee to 0.001 nmc. How does that stop them getting re-broadcast?

[ryanc]

Some points:

* This party did not pay the minimum fee, on their transactions, which is 0.005 * ceil(tx_size/1000) NMC - most of the offending transactions were > 16000 bytes and should have paid 0.085 NMC. The transactions should not have propagated, but MIN_RELAY_TX_FEE was set to MIN_TX_FEE/50 - on bitcoin it is MIN_TX_FEE/5. The 'bandaid' patch fixes that.

* According to petertodd the mempool size isn't bounded even on bitcoin. It is probably the case that doing something similar - sending a tremendous volume of standard transactions that have a large enough fee to be relayed but not big enough to be mined quickly - would screw up other coins, especially those that are merge mined.

* We should see if we can fix the performance of getmemorypool and getauxblock in cases where the mempool has a large volume of transactions in it.

[indolering]

* According to petertodd the mempool size isn't bounded even on bitcoin. It is probably the case that doing something similar - sending a tremendous volume of standard transactions that have a large enough fee to be relayed but not big enough to be mined quickly - would screw up other coins, especially those that are merge mined.

You are saying that BTC might be vulnerable as well? Could someone pull them in on this? It's much easier to get Bitcoin devs to fix it.

[phelix]

* We should see if we can fix the performance of getmemorypool and getauxblock in cases where the mempool has a large volume of transactions in it.

this. And/or limit the amount of work that is done on prioritizing and assembling the block.

[pmc]

* According to petertodd the mempool size isn't bounded even on bitcoin. It is probably the case that doing something similar - sending a tremendous volume of standard transactions that have a large enough fee to be relayed but not big enough to be mined quickly - would screw up other coins, especially those that are merge mined.

I did something similar on sexcoin a couple of months ago, but on a much smaller scale. I noticed that the block time went up from the standard 1 minute to >10 and I was getting a little nervous because I thought I'd killed the network, but it quickly recovered - after a few blocks my transactions were through and everything was back to normal.

One thing to note here is that I used a perl script + RPC API to make sure my transactions stayed below 1 kB and used zero fees. So the suggested band-aid patch would not have worked against my involuntary attack.

Is the "great aggregation" really a problem (as in breaking things), or is it only slowing down the network? In the latter case I'd rather scrap the "band-aid" patch and concentrate on finding and fixing the performance issue.

[sudoquai]

Is the "great aggregation" really a problem (as in breaking things), or is it only slowing down the network? In the latter case I'd rather scrap the "band-aid" patch and concentrate on finding and fixing the performance issue.

Good point, the "band-aid" patch sounds more like a workaround for me, imho.

[indolering]

Is the "great aggregation" really a problem (as in breaking things), or is it only slowing down the network? In the latter case I'd rather scrap the "band-aid" patch and concentrate on finding and fixing the performance issue.

Good point, the "band-aid" patch sounds more like a workaround for me, imho.

Please refrain from "me too" posts.