Concerns about removing BDB lock limit

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Concerns about removing BDB lock limit

Post by biolizard89 »

This commit in Daniel's proposed AlwaysAuxpowActive hardfork eliminates the BDB lock limit. https://github.com/domob1812/namecore/c ... 89741d309c

I'm posting this thread to raise concerns about this change.

First off, some background on block size tradeoffs. This is transcribed from a slide in Adam Back's presentation "Bitcoin Scaling Tradeoffs":
Adam Back's slides wrote: Why do miners prefer smaller blocks?
  • Orphan rate - mining is a race to propagate block in last 3 seconds
  • Limit is latency, burstiness not bandwidth
  • 1MB is not much, but it is hard to global broadcast in 3 seconds
Workarounds today
  • Use a pool! (eg slush!)
  • Relay network (custom routes, network compression, much faster medium size miners)
  • Peering (large miners use peering)
  • Not so good: validationless mining reduces SPV security (eg BIP66 4th july side-effect)
What happens if orphan rate goes up?
  • Miners work around it: more validationless mining, more centralized huge pools... one pool?
  • But centralisation presents risk to Bitcoin's main properties.
Second, some info from Luke-Jr via #namecoin-dev about what block size is safe if we don't want the above "workarounds" to happen.

(Some messages have been redacted due to being off-topic.)

Code: Select all

Apr 21 04:14:11 <Jeremy_Rand>	Luke-Jr: are my recollections of 3 to 4MB being the "safe" limit for block size with current relay tech still accurate?  I haven't read up on this topic in many months
Apr 21 04:15:13 <qpm>	freenode:<Luke-Jr> Jeremy_Rand: safe is like 400-500k :/
Apr 21 04:15:22 <qpm>	freenode:<Luke-Jr> maybe lower, I forgt
Apr 21 04:16:02 <Jeremy_Rand>	Luke-Jr: ah.  That's news to me.  I could have sworn someone (was it Austin Hill?) said in a press interview the limit was between 3 and 4?
Apr 21 04:16:03 <qpm>	freenode:<Squidicuz> Luke-Jr, so we should be mining smaller blocks at the moment?
Apr 21 04:16:22 <qpm>	freenode:<Luke-Jr> Squidicuz: yes
Apr 21 04:16:28 <qpm>	freenode:<Squidicuz> okay :\
Apr 21 04:16:36 <Jeremy_Rand>	(I'm not arguing with you, just curious why my memory conflicts with what you say)
Apr 21 04:16:37 <qpm>	freenode:<Luke-Jr> Jeremy_Rand: for a centralised Bitcoin, that might be true
Apr 21 04:16:47 <Jeremy_Rand>	ah
Apr 21 04:16:48 <Jeremy_Rand>	I see
Apr 21 04:16:51 <qpm>	freenode:<Luke-Jr> right now, Bitcoin is very centralised :/
Apr 21 04:17:05 <Jeremy_Rand>	yeah.  Centralization of Bitcoin makes me sad
Apr 21 04:17:09 <qpm>	freenode:<Squidicuz> centralised where though?
Apr 21 04:17:51 <qpm>	freenode:<Squidicuz> which aspect*
Apr 21 04:18:05 <qpm>	freenode:<Luke-Jr> Squidicuz: there's handful of miners, who rely on a block relay backbone run by Matt
Apr 21 04:18:22 <qpm>	freenode:<Squidicuz> true, but why is that bad?
Apr 21 04:18:54 <qpm>	freenode:<Squidicuz> the whole network is made of relying on someone who is doing something for someone/something
Apr 21 04:18:56 <qpm>	freenode:<Squidicuz> ...
Apr 21 04:19:29 <Jeremy_Rand>	Luke-Jr: curious why the block relay system from Matt can't be used for the public P2P network?
Apr 21 04:19:42 <qpm>	freenode:<Squidicuz> it is isolated collusion between pockets that breed corruptions :s
Apr 21 04:20:05 <qpm>	freenode:<Luke-Jr> Jeremy_Rand: the benefit is mostly because there's 2 hops instead of N
Apr 21 04:20:07 <qpm>	freenode:<Squidicuz> Jeremy_Rand, in latest there is part of that relay for just the blocks
Apr 21 04:20:18 <qpm>	freenode:<Squidicuz> _i think_
Apr 21 04:20:32 <Jeremy_Rand>	Luke-Jr: oh.  So it only works because it's centralized by design.
Apr 21 04:20:40 <qpm>	freenode:<Luke-Jr> yes
Apr 21 04:20:52 <Jeremy_Rand>	so many things on the Internet have that property
Apr 21 04:20:53 <qpm>	freenode:<Luke-Jr> I mean, Matt is working on new block relay p2p code, but that can only do so much
Apr 21 04:20:54 <qpm>	freenode:<Squidicuz> oh
Apr 21 04:20:56 <Jeremy_Rand>	makes me sad
Apr 21 04:21:01 <qpm>	freenode:<Squidicuz> but that is and isn;'t bad. .
Apr 21 04:21:22 <qpm>	freenode:<Luke-Jr> Squidicuz: a court could literally order Matt to censor an address, and he could effectively do it
Apr 21 04:21:24 <qpm>	freenode:<Squidicuz> it could be much better. yes
Apr 21 04:21:37 <qpm>	freenode:<Squidicuz> hmmm
Apr 21 04:21:47 <qpm>	freenode:<Luke-Jr> any block with a transaction paying that address would be penalised heavily by the relay network filtering it
Apr 21 04:21:55 <qpm>	freenode:<Squidicuz> Luke-Jr, cuz the service is centralized. ?
Apr 21 04:22:06 <qpm>	freenode:<Luke-Jr> Squidicuz: yes
Apr 21 04:22:13 <qpm>	freenode:<Squidicuz> hmph
Apr 21 04:22:33 <qpm>	freenode:<Squidicuz> Luke-Jr, but isn't that block relay part being added to mian?
Apr 21 04:22:43 <qpm>	freenode:<Squidicuz> I mean core.. err soon*
Apr 21 04:22:58 <qpm>	freenode:<Luke-Jr> the protocol improvements are, but you can't reduce hops in p2p
I checked the Namecoin block explorer on webbtc. Right now, only 7% of the last 100 blocks are over 10 kbytes. The largest block in the last 100 blocks was 40.6 kbytes.

The BDB lock limit, according to https://github.com/bitcoin/bips/blob/ma ... .mediawiki , in practice enforces a maximum block size that is a bit larger than 500 kbytes, which is close to the threshold that Luke describes as safe. It's also much larger than our currently mined blocks. I asked Luke on #namecoin-dev whether it would be fair to infer from this that we should keep the BDB lock limit in place.

(Some messages have been redacted due to being off-topic.)

Code: Select all

May 15 18:14:34 <Jeremy_Rand>	Hi Luke-Jr.  On April 21 you said here that the safe block size limit for a decentralized Bitcoin would be somewhere around 400-500k?  Any chance you might be able to provide me with a link where I could read up on that?
May 15 18:32:22 <qpm>	freenode:<luke-jr> nope
May 15 18:36:51 <Jeremy_Rand>	Luke-Jr: let me give you some background on why I'm curious about the 400-500k.  Right now Namecoin has a lower block size limit than Bitcoin, because we never lifted the BDB lock limit (ported from the temporary fix that Bitcoin Core deployed before BDB was phased out)
May 15 18:37:52 <Jeremy_Rand>	At the moment, domob is planning to remove that lock limit when we hardfork for BIP9.  But if lower block size than Bitcoin's current max is beneficial, that might affect our decision on whether to leave the lock limit in place.
May 15 18:39:19 <Jeremy_Rand>	So, if you're able to provide us with some info so that we can make a more informed decision on that, it would be greatly appreciated
May 15 18:40:38 <qpm>	freenode:<luke-jr> as of right now, mining over the p2p network is unfeasible
May 15 18:40:45 <qpm>	freenode:<luke-jr> miners all rely on Matt's centralised block relay backbone
May 15 18:46:24 <Jeremy_Rand>	luke-jr: would that situation be improved if the block size were limited to the amounts that the BDB lock limit enforced?  If it would be an improvement, would it be slight or significant?
May 15 18:47:04 <qpm>	freenode:<luke-jr> it'd be an accidental improvement in the typical case
May 15 18:47:25 <qpm>	freenode:<luke-jr> that being said, the bdb lock limit is arguably a useful limit to retain anyway
May 15 18:49:14 <Jeremy_Rand>	luke-jr: ok.  So in Namecoin's case, since we don't have a huge volume of non-spam transactions, and since we value decentralization highly, would it make sense for us to keep the BDB lock limit in place, at least until either we seriously need more block size or better scaling tech is produced in the Bitcoin world?
May 15 18:49:50 <qpm>	freenode:<luke-jr> probably
May 15 18:51:21 <Jeremy_Rand>	luke-jr: ok.  Thanks for the analysis.  Is it okay if I send a chatlog of this conversation to some other Namecoin devs who don't hang around in IRC all the time?  (Or for that matter, is it okay if I paste it into the public Namecoin forum?)
May 15 18:51:37 <qpm>	freenode:<luke-jr> sure
May 15 18:52:10 <Jeremy_Rand>	is that a "sure" to both requests, or just the former?
May 15 18:52:18 <qpm>	freenode:<luke-jr> I don't care
May 15 18:52:24 <qpm>	freenode:<luke-jr> I assume this channle is public anyway
May 15 18:53:04 <Jeremy_Rand>	ok.  I'll post it on the Namecoin forum when I have a few minutes
So, it seems that we should carefully consider whether the BDB lock limit should be removed as part of the AlwaysAuxpowActive hardfork. It is my opinion that we don't need the increased capacity anytime soon, and we benefit from the decentralization that comes with the BDB lock limit. There will be plenty of times when we're likely to do hardforks later (there are 3 other AuxPoW-related hardforks that are proposed), so we can roll a removal of the BDB lock limit into one of those if it's needed then. There are other scaling-related changes in the pipeline that can make increased capacity somewhat safer, e.g. more compact encoding of name values via CBOR and compression (Hugo and I have been discussing this a bit).

Thoughts on this?
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

johnc
Posts: 89
Joined: Sun Dec 28, 2014 10:03 am

Re: Concerns about removing BDB lock limit

Post by johnc »

I think this is the same bitcoin block size debate etc. i don't think we should implement another arbitrary limit to the growth of namecoin if it's going to require a hard fork to remove it later.

First of all, let me state that i also want to be able to run namecoin on my laptop but is not like namecoin is going to reach a bitcoin transaction level any time soon.

However if you want to talk about decentralization, I will be more concerned about why not all bitcoin mining pools are mining namecoin and thus having one pool control the majority of hash power.

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Concerns about removing BDB lock limit

Post by biolizard89 »

johnc wrote:I think this is the same bitcoin block size debate etc. i don't think we should implement another arbitrary limit to the growth of namecoin if it's going to require a hard fork to remove it later.

First of all, let me state that i also want to be able to run namecoin on my laptop but is not like namecoin is going to reach a bitcoin transaction level any time soon.

However if you want to talk about decentralization, I will be more concerned about why not all bitcoin mining pools are mining namecoin and thus having one pool control the majority of hash power.
I am delighted to see that the toxic block size trolls have found their way here from Bitcoinland. First that dude on /r/Namecoin who harassed me and Brandon over Brandon's GUI PR, now this.

I don't think it's even worth doing a line-by-line rebuttal to prove that you're speaking nonsense, seeing as you're transparently proving that all by yourself. So instead I will just say that the laws of physics don't change just because you don't want to understand them, and you should either make an argument based in engineering, or let the engineers do the engineering. Don't hijack this thread with further irrelevant ramblings, or said ramblings will be removed.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

johnc
Posts: 89
Joined: Sun Dec 28, 2014 10:03 am

Re: Concerns about removing BDB lock limit

Post by johnc »

maybe i'm speaking nonsense because i don't understand what is the meaning of "BDB locksize limit". I just understand "there's a limit and we don't want to raise it". Is because spam? Maybe i have been mislead by the r/bitcoin trolls. However my question still stand:

Why namecoin should be more limited than bitcoin?.

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Concerns about removing BDB lock limit

Post by biolizard89 »

johnc wrote:maybe i'm speaking nonsense because i don't understand what is the meaning of "BDB locksize limit". I just understand "there's a limit and we don't want to raise it". Is because spam? Maybe i have been mislead by the r/bitcoin trolls. However my question still stand:

Why namecoin should be more limited than bitcoin?.
The BDB lock limit is a limit on the number of database operations a block can contain. (For example, inputs, outputs, and name operations count toward this limit.) It is loosely correlated to the block size, although the relation between the two depends on what kind of transactions are in the blocks. Bitcoin hardforked a few years ago (I think 2013) to remove this limit. It arguably made sense for Bitcoin because they were coming close to the limit, which constrained growth. Unfortunately, the laws of physics (mainly the speed of light) cause a tradeoff between block size and orphan rate. Larger blocks cause more orphans, which causes more centralized mining. As Luke stated, the really lousy situation that Bitcoin is in right now (Matt being able to censor transactions if he wants) is because their blocks are too big.

Bitcoin can't easily do anything about this, because they actually do have a large transaction volume right now, which can't wait for better scalability tech. We, however, do have that luxury, because our blocks are not anywhere near the lock limit. If, somehow, our transaction volume increases 100-fold or so (that's a rough number) and we get close to the lock limit, then the only immediate effect would be transaction fees going up a bit. Guess what that would mean? Less squatting. That's not a bad thing at all.

It's important to note that this isn't about adding a new limit. This is a limit that's already there, that I'm saying we don't have a good reason to remove. It's also important to note that AlwaysAuxpowActive is not the last hardfork we'll do. The P2Pool AuxPoW hardfork, the SegWit AuxPoW hardfork, and the Blockstream AuxPoW hardfork are all proposed. Bitcoin is planning their own hardfork for next year (with various technical improvements), which we're likely to merge as well. And then there are proposals like dynamic name pricing, which would possibly be a hardfork. Any of these planned hardforks could also remove the lock limit (which places us at a 1 MB block size limit just like Bitcoin is now) if there were indication that it might be needed soon. Given the exponential curve of Bitcoin's growth, and where we are on that curve, we would have a lot of advance warning if we were approaching the lock limit, particularly given that legitimate users are usually more willing to pay a moderately large fee on domain name registrations/renewals than for Bitcoin transactions.

Finally, note that since Chinese pools have a majority of hashrate, and intra-China latency is much better than latency for traffic crossing the Chinese firewall, if Chinese pools wanted to cause mischief on the Namecoin network, they could intentionally fill up blocks with spam transactions in order to increase the orphan rate of non-Chinese pools. This would be profitable for Chinese pools, and give them a larger fraction of the blocks mined. While I'm confident that F2Pool isn't going to do this (Wang Chun is a good guy, he's not likely to do such a thing), it is not a good idea to open that attack vector when we don't need the only benefits that are associated with it.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: Concerns about removing BDB lock limit

Post by domob »

If there are serious concerns about removing the limit, then we can of course not do it for now. However, my feeling is that it is not necessary to have it. Until you get some dependable modelling, I wouldn't really believe that the current limit is just good while the full blocksize limit is too large. Instead, if there are real concerns over the size limit (whatever the safe value is), we should remove the hackish lock limit and instead lower the blocksize hard limit. That's way more transparent, clean and safe.

(Note that this is independent of what the actual target size limit would be. My own opinion is that we should go for Bitcoin's 1 MiB limit, unless we get a convincing calculation that clearly shows that some other limit is better.)
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Concerns about removing BDB lock limit

Post by biolizard89 »

domob wrote:If there are serious concerns about removing the limit, then we can of course not do it for now. However, my feeling is that it is not necessary to have it. Until you get some dependable modelling, I wouldn't really believe that the current limit is just good while the full blocksize limit is too large. Instead, if there are real concerns over the size limit (whatever the safe value is), we should remove the hackish lock limit and instead lower the blocksize hard limit. That's way more transparent, clean and safe.

(Note that this is independent of what the actual target size limit would be. My own opinion is that we should go for Bitcoin's 1 MiB limit, unless we get a convincing calculation that clearly shows that some other limit is better.)
I'm aware that Blockstream has done modeling/experiments of various block sizes' safety. I don't know if that data is published; I'll ask Luke. From the single data point of right now, Bitcoin's blocks are close to 1 MB, and miners are currently doing the "workarounds" that Adam and Luke described, e.g. Matt's relay network.

I'm curious exactly when the relay network became popular, and what Bitcoin's typical block size was at that time. I don't know how much data is available about the relay network's historical usage. I'm also curious exactly how much mining diversity has changed over time, both by pool and by jurisdiction, and how that correlates with historical block size.

My guess is that Blockstream already has looked at all this data, and probably a lot of other data, which is presumably why Luke was fairly confident that 1 MB wasn't safe if decentralization is a goal.

If we had good data and analysis on a "safe" block size that had been widely reviewed, I'd probably be in favor of removing the BDB lock limit and changing the block size limit accordingly (although I'd want to sleep on it before being definitely in favor). However, I am doubtful that we have the resources to fully evaluate this topic without delaying AlwaysAuxpowActive. Given that AlwaysAuxpowActive needs to be done as soon as reasonably feasible, and that rolling a potentially controversial change into a consensus fork that needs to be uncontroversially adopted isn't really a great thing, I think what I favor is postponing any changes to the lock limit or the block size limit until after AlwaysAuxpowActive is activated, i.e. AAA wouldn't change the lock limit or block size limit.

Unlike Bitcoin, we're not facing a scaling crisis, so we can afford to take our time on this one -- the status quo seems to be acceptable for the short term future.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: Concerns about removing BDB lock limit

Post by domob »

My concern is just that the BDB lock limit is actually a very hackish thing, especially when viewed as a limit to the block size. Thus it would be much nicer to get rid of it and, if necessary, set an explicit limit (e. g. at the value that the lock limit "articifically" enforces). But let's see what others say here, I guess - I'm not absolutely against leaving the limit if it makes the fork easier.
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

cassini
Posts: 336
Joined: Sun May 26, 2013 6:36 pm

Re: Concerns about removing BDB lock limit

Post by cassini »

I'm trying to figure out the consequences of a few edge scenarios that may occur in the future.
The Great Aggregator of August 2014 is the one that immediately springs to mind.
ATM I can't remember if the blocks hit the BDB lock limit or not. The only technical detail I can remember is that the old 0.3.x clients at the mining pools spent too much time with validating these Great Aggregator blocks. The clients either crashed or were simply too busy validating and therefore couldn't keep up with the average 10-mins. blockrate.
Thanks to Namecoin Core and its much faster validation algorithm a similar Great Aggregator event can't harm the network anymore. What would this situation look like if we removed the BDB lock limit?

hla
Posts: 46
Joined: Mon Nov 10, 2014 12:01 am
os: linux
Contact:

Re: Concerns about removing BDB lock limit

Post by hla »

The lock limit is a bug and should be fixed. The block size limit for Namecoin, as for Bitcoin is 1M. The lock limit is a bug in one implementation of Namecoin, causing it not to comply with the parameters of the network. It should be fixed. If you think the block size limit for Namecoin should be reduced, that should be argued for and effected in its own right.

The lock limit is an artefact of a bug in a specific implementation and its precise behaviour is as such characterised by the innards of that implementation. It is highly inappropriate to use it as a defined limitation on a cryptocurrency. If you want to make it a formal limit, how are other Namecoin node implementations, which do not necessarily use BDB, supposed to effect it? Presumably they would have to calculate the exact number of database operations which a BDB-based node, no, a specifically Core-based node, would have done in order to effect that limit. That's absurd, and a design disaster. It's certainly not something that should be effected deliberately, even temporarily.

Post Reply