Scalability of namecoin

Post Reply
rystsov
Posts: 1
Joined: Mon Dec 09, 2013 6:59 pm
Contact:

Scalability of namecoin

Post by rystsov »

It seems that an idea to store all information inside a blockchain would lead to its bloating.

There are about 250MM of domains in the web, namecoin allows to store 1Kb of data per record, it leads to 250Gb of data to store in the blockchain, but namecoin is far more general key/value storage than DNS, so the real size of blockchain is expected to be larger.

Am I right? What is the strategy to avoid or to deal with this bloating?

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: Scalability of namecoin

Post by phelix »

rystsov wrote:It seems that an idea to store all information inside a blockchain would lead to its bloating.

There are about 250MM of domains in the web, namecoin allows to store 1Kb of data per record, it leads to 250Gb of data to store in the blockchain, but namecoin is far more general key/value storage than DNS, so the real size of blockchain is expected to be larger.

Am I right? What is the strategy to avoid or to deal with this bloating?
Bitcoin has to deal with more TXs than Namecoin so if they make it we should be able to do so, too.

IMHO 250 milllion is more than what is necessary for domains. Other applications could be splitted to separate blockchains should it be deemed necessary.

Also 250GB is not so much especially for a server. I can buy a TB for 50€ today. In a couple of years I will probably have that on my smartphone.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Scalability of namecoin

Post by biolizard89 »

It is unlikely that most domains will use 1KiB. Right now the limit is ~520B, and almost no sites are even close to that. It would be interesting if someone wanted to do a sampling of valid .bit sites in the blockchain and see what the value length distribution is.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

indolering
Posts: 800
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: Scalability of namecoin

Post by indolering »

For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.

Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow :)
DNS is much more than a key->value datastore.

biolizard89
Posts: 2001
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Scalability of namecoin

Post by biolizard89 »

indolering wrote:For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.

Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow :)
Nice, that's interesting work. It would be interesting to see proposals on storage optimizations... I think gzip and binary JSON were discussed in passing years ago, but to my knowledge no one has worked on this as long as I've been active here (about 1.5 years).
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

indolering
Posts: 800
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: Scalability of namecoin

Post by indolering »

I looked into data structures a while back and there were some interesting IP address compression structs, but that's all I can remember. At this point I think "optimizing" the format is premature optimization, JSON is convenient and text is cheap!
DNS is much more than a key->value datastore.

Post Reply