It seems that an idea to store all information inside a blockchain would lead to its bloating.
There are about 250MM of domains in the web, namecoin allows to store 1Kb of data per record, it leads to 250Gb of data to store in the blockchain, but namecoin is far more general key/value storage than DNS, so the real size of blockchain is expected to be larger.
Am I right? What is the strategy to avoid or to deal with this bloating?
Scalability of namecoin
Re: Scalability of namecoin
Bitcoin has to deal with more TXs than Namecoin so if they make it we should be able to do so, too.rystsov wrote:It seems that an idea to store all information inside a blockchain would lead to its bloating.
There are about 250MM of domains in the web, namecoin allows to store 1Kb of data per record, it leads to 250Gb of data to store in the blockchain, but namecoin is far more general key/value storage than DNS, so the real size of blockchain is expected to be larger.
Am I right? What is the strategy to avoid or to deal with this bloating?
IMHO 250 milllion is more than what is necessary for domains. Other applications could be splitted to separate blockchains should it be deemed necessary.
Also 250GB is not so much especially for a server. I can buy a TB for 50€ today. In a couple of years I will probably have that on my smartphone.
-
- Posts: 2001
- Joined: Tue Jun 05, 2012 6:25 am
- os: linux
Re: Scalability of namecoin
It is unlikely that most domains will use 1KiB. Right now the limit is ~520B, and almost no sites are even close to that. It would be interesting if someone wanted to do a sampling of valid .bit sites in the blockchain and see what the value length distribution is.
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: Scalability of namecoin
For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.
Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow
Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow
DNS is much more than a key->value datastore.
-
- Posts: 2001
- Joined: Tue Jun 05, 2012 6:25 am
- os: linux
Re: Scalability of namecoin
Nice, that's interesting work. It would be interesting to see proposals on storage optimizations... I think gzip and binary JSON were discussed in passing years ago, but to my knowledge no one has worked on this as long as I've been active here (about 1.5 years).indolering wrote:For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.
Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: Scalability of namecoin
I looked into data structures a while back and there were some interesting IP address compression structs, but that's all I can remember. At this point I think "optimizing" the format is premature optimization, JSON is convenient and text is cheap!
DNS is much more than a key->value datastore.