Reply to topic  [ 12 posts ]  Go to page 1, 2  Next
Bit domains statistics 
Author Message

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Bit domains statistics
I`am creating separate topic for subj, because subj started in other topic "DNS server for .bit".
biolizard89 wrote:
virus_net wrote:
Now I`am working at domain names statistics script. I want to do graph that can show how many domain names was at that date, how many of them was expired at that date and how many of them resolving/working.
Because for today .bit have 72320 domain names but we can`t understand now how many of them really used/working.

This would indeed be useful information to have. In related work, Jonas has been doing some experiments with making the YaCy search engine crawl .bit websites, which would also give us useful information on how much real-world usage there is of .bit.

So, I wrote script that get data by command name_filter with regexp ^d/[a-z]([a-z0-9-]{0,62}[a-z0-9])?$.
Results are awful :shock: For today:
    Total: 72408 domains
    Null value field: 31179 domains
    Value field JSON parse error: 16079 domains
    No useful data found (ip or ns or alias): 23087 domains
    Usefull data found (ip or ns or alias): 2063 domains

So only 2063 domains from 72408 has usefull data and can be tried for resolv and access. So 70345 domains are useless...
If someone interested in debug log of the script it is avaliable here: https://bitname.ru/tmp/name_scan_nf.txt

I think the best data in value field are these:
peeper.bit wrote:
I'm Plasmmer. Plasmmer is my nickname. I was born with the full name Daniell Wilson Jose Mesquita, on the city of Cascavel, state of Parana, in Brazil. I was born in April 9 of the 1997 year, and currently I'm 20 years old. I declare I'm owner of this Namecoin address, of this domain (peeper.bit), of any copyrights/registers I do by using file hash codes in transactions and of any files registered here.

cryptostamp.bit wrote:
on that subject, here are two hashes I may later use to prove some party with losses was not related to this: 48be8a34e1a972b7299f52d11ec5c1a68070b5be5f342a997657b385e1f73830 282af96c48c3ed5864b8e5a521452bbca9f8b7b85b8ddf6307415e2d11b67f05 e7ab179d9020e2ab1800853e504a43ef2c7dd217740a00909c98218044123b6b


After this:
Quote:
Value field JSON parse error: 16079 domains

one question is coming up, question to the developers:
Why there is no value field cheking before sending it to the chain ?

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Last edited by virus_net on Thu Oct 19, 2017 1:08 pm, edited 1 time in total.



Thu Oct 19, 2017 12:40 pm
Profile WWW

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Re: Bit domains statistics
Next I want to take domains that have useful data and try to:
    - resolv
    - access HTTP or HTTPS
And then count again, count how many domains are really working.

What do you think about how much of 2063 will remain ? Make your bets gentlemen :)

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Thu Oct 19, 2017 12:43 pm
Profile WWW

Posts: 1927
os: linux
Reply with quote
Post Re: Bit domains statistics
virus_net wrote:
So, I wrote script that get data by command name_filter with regexp ^d/[a-z]([a-z0-9-]{0,62}[a-z0-9])?$.
Results are awful :shock: For today:
    Total: 72408 domains
    Null value field: 31179 domains
    Value field JSON parse error: 16079 domains
    No useful data found (ip or ns or alias): 23087 domains
    Usefull data found (ip or ns or alias): 2063 domains

So only 2063 domains from 72408 has usefull data and can be tried for resolv and access. So 70345 domains are useless...
If someone interested in debug log of the script it is avaliable here: https://bitname.ru/tmp/name_scan_nf.txt


Those numbers look reasonably close to what I'd expect.

virus_net wrote:
I think the best data in value field are these:
peeper.bit wrote:
I'm Plasmmer. Plasmmer is my nickname. I was born with the full name Daniell Wilson Jose Mesquita, on the city of Cascavel, state of Parana, in Brazil. I was born in April 9 of the 1997 year, and currently I'm 20 years old. I declare I'm owner of this Namecoin address, of this domain (peeper.bit), of any copyrights/registers I do by using file hash codes in transactions and of any files registered here.


Someone mentioned this name on Matrix a few weeks ago. That value appears in multiple names. I would not be surprised if this is actually someone using the Namecoin blockchain to doxx Mesquita.

virus_net wrote:
cryptostamp.bit wrote:
on that subject, here are two hashes I may later use to prove some party with losses was not related to this: 48be8a34e1a972b7299f52d11ec5c1a68070b5be5f342a997657b385e1f73830 282af96c48c3ed5864b8e5a521452bbca9f8b7b85b8ddf6307415e2d11b67f05 e7ab179d9020e2ab1800853e504a43ef2c7dd217740a00909c98218044123b6b


I really wish people would use OpenTimestamps for timestamping instead of trying to use Namecoin for it. Namecoin isn't designed for this use case and such usage doesn't scale.

virus_net wrote:
After this:
Quote:
Value field JSON parse error: 16079 domains

one question is coming up, question to the developers:
Why there is no value field cheking before sending it to the chain ?


There actually used to be a JSON validity check in the GUI, although the 0.13.99-name-tab-beta1 release doesn't have that (it will presumably be re-added later). I suspect that most of the non-JSON values are intentional (e.g. if someone uses a Bitmessage address as the value). Technically Bitmessage addresses should be placed in the "info" field, but for some reason most people don't do that.

Anyway you are right, we should add validity checks to the GUI. (Maybe the RPC interface too, although generally I figure people who are using the RPC interface are capable of doing those checks themselves.)

virus_net wrote:
What do you think about how much of 2063 will remain ? Make your bets gentlemen :)


Guessing somewhere in the vicinity of 100 for HTTP, but I'm not particularly confident in this estimate. For HTTPS, I'd estimate fewer than 5 will have a valid certificate, since we rolled out TLS cert support very recently and I don't think it's gotten much adoption yet. Maybe I'll be surprised.

_________________
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5


Fri Oct 20, 2017 6:35 pm
Profile

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Re: Bit domains statistics
This is also interesting stat:
Code:
#cat name_scan.log | grep sale | wc -l
   15611

So 15611 domains have word "sale" in value field.

biolizard89 wrote:
Anyway you are right, we should add validity checks to the GUI.

It will be certainly good.
biolizard89 wrote:
although generally I figure people who are using the RPC interface are capable of doing those checks themselves.

You are 50% right. Why 50 ? Because people are lazy now and people always doing mistakes (copy-paste), it`s people nature :)
I think that checks will be good everywhere. Strictness of verification maybe different. In PRC only validation of if it`s a valid JSON object and in GUI plus object data validation.
biolizard89 wrote:
Guessing somewhere ...

Your bet accepted :)

P.S. From yesterday this data is avail on the first graph here.
Blue line is total.
Green line is Null value + JSON parse error + No useful data found.
Black one is Usefull data found.

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Sat Oct 21, 2017 4:47 am
Profile WWW

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Re: Bit domains statistics
virus_net wrote:
count how many domains are really working.

Done.
So for today values are:
    Total: 72492 domains
    Null value field: 31160 domains
    Value field JSON parse error: 16065 domains
    No useful data found (ip or ns or alias): 23196 domains
    Usefull data found (ip or ns or alias): 2071 domains
    Really working (responded on HTTP request): 1108 domains
So it`s a half. But 725 domains from them have "parked at dotbit.me." page. So I will exclude them too.

Now you can easily get list of all .bit domains at this page.
First checkbox is for view all domains.
Second checkbox is for view only that domains which have DNS data.
Third checkbox is for view only that domains which responded on HTTP request and don`t parked at dotbit.me.

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Sun Oct 22, 2017 8:12 am
Profile WWW

Posts: 1927
os: linux
Reply with quote
Post Re: Bit domains statistics
virus_net wrote:
This is also interesting stat:
Code:
#cat name_scan.log | grep sale | wc -l
   15611

So 15611 domains have word "sale" in value field.


That's actually more than I would have expected; usually when I see a name that's for sale, the value just consists of a Bitmessage address (under the assumption that whoever wants the name will message the owner on Bitmessage, and find out from there that the name is for sale). Any chance you could give a sample of a few such names? I'm curious what their values look like.

virus_net wrote:
biolizard89 wrote:
although generally I figure people who are using the RPC interface are capable of doing those checks themselves.

You are 50% right. Why 50 ? Because people are lazy now and people always doing mistakes (copy-paste), it`s people nature :)
I think that checks will be good everywhere. Strictness of verification maybe different. In PRC only validation of if it`s a valid JSON object and in GUI plus object data validation.


It should be noted that JSON encoding is solely a convention in Namecoin (Namecoin's consensus rules simply treat the name and value fields as arbitrary binary blobs). Also Namecoin might migrate in the future from JSON to CBOR as the standard convention (due to CBOR's decreased tendency to bloat the blockchain). That said, I can definitely see some value add in making the RPC interface check for JSON validity, as long as the check is easy to disable for people who know what they're doing (or who think they do, anyway :) ). Thanks for the suggestion, I've added a GitHub issue for it at https://github.com/namecoin/namecoin-core/issues/191

_________________
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5


Tue Oct 24, 2017 7:39 am
Profile

Posts: 1927
os: linux
Reply with quote
Post Re: Bit domains statistics
virus_net wrote:
virus_net wrote:
count how many domains are really working.

Done.
So for today values are:
    Total: 72492 domains
    Null value field: 31160 domains
    Value field JSON parse error: 16065 domains
    No useful data found (ip or ns or alias): 23196 domains
    Usefull data found (ip or ns or alias): 2071 domains
    Really working (responded on HTTP request): 1108 domains
So it`s a half. But 725 domains from them have "parked at dotbit.me." page. So I will exclude them too.


So, 1108 - 725 = 383 .bit domains that reply on HTTP with something other than "parked at dotbit.me"? Nice, that's quite a lot more than I was expecting.

virus_net wrote:
Now you can easily get list of all .bit domains at this page.
First checkbox is for view all domains.
Second checkbox is for view only that domains which have DNS data.
Third checkbox is for view only that domains which responded on HTTP request and don`t parked at dotbit.me.


Cool, I'll have to take a look through this list and see what these domains are being used for. Most of them I've never heard of. (Of course, since it's possible that a few of the domains are being used for malware, I don't recommend that anyone reading this thread visit random domains from the list unless they're doing so in a VM.)

It should be noted that you're undercounting the full set of "in-use" .bit domains, because some domains use JSON attributes that you're not currently supporting (e.g. domob.bit uses "import", which works in ncdns but I guess doesn't work for your software). There are also some .bit domains that use non-DNS resolution, e.g. Tor onion services or ZeroNet. So I think the actual number of in-use .bit domains is going to be noticeably higher than your estimate of 383.

Anyway, excellent work, this will give me something to play with next weekend. :)

_________________
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5


Tue Oct 24, 2017 7:50 am
Profile

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Re: Bit domains statistics
biolizard89 wrote:
usually when I see a name that's for sale, the value just consists of a Bitmessage address

I don`t agree with that. Remember bitname.bit story ? ;)
biolizard89 wrote:
Any chance you could give a sample of a few such names?

Ofc I can. Here you are:
Code:
a0.bit {"for_sale" : 1, "website" : "http://blockchained.com", "name" : "phelix"}
bf.bit salenamecoin@gmail.com
zyv.bit{"map":{"":"10.0.0.1"},"email":"nmc.name.for.sale@gmail.com","import":"d/readme-nameforsale"}

Also you can download full log and grep as me:
virus_net wrote:
If someone interested in debug log of the script it is avaliable here: https://bitname.ru/tmp/name_scan_nf.txt

I replace file and put there debug from today.
biolizard89 wrote:
I've added a GitHub issue for it

I saw. Thanks.
biolizard89 wrote:
So, 1108 - 725 = 383 .bit domains that reply on HTTP with something other than "parked at dotbit.me"?

Yes. I excluded "parked at dotbit.me" domains from the "working" statistics.
biolizard89 wrote:
Cool, I'll have to take a look through this list and see what these domains are being used for. Most of them I've never heard of.

This was my goal. To see list of domains that can be opened and not full list (as example) and you need to click all of them to find working one.
biolizard89 wrote:
It should be noted that you're undercounting the full set of "in-use" .bit domains

You right. But for now it is not so important because there are not so much and I don`t know how to test them yet. It`s in my TODO list.
For now, if you wish, I can put such domains in "in-use" and "working" without checking.
biolizard89 wrote:
Anyway, excellent work, this will give me something to play with next weekend.

Thanks. You welcome :)

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Tue Oct 24, 2017 8:04 am
Profile WWW

Posts: 1927
os: linux
Reply with quote
Post Re: Bit domains statistics
virus_net wrote:
biolizard89 wrote:
usually when I see a name that's for sale, the value just consists of a Bitmessage address

I don`t agree with that. Remember bitname.bit story ? ;)


I didn't intend for the "when" operator in my comment to be commutative. :) There are definitely people who only list a Bitmessage address, who are happy to give away the name for free. I would expect, though, that most of the people who are trying to sell a name would also list only a Bitmessage address. That said, I don't have any empirical evidence to back this assumption up, since I've never tried to purchase a name.

virus_net wrote:
biolizard89 wrote:
Anyway, excellent work, this will give me something to play with next weekend.

Thanks. You welcome :)


Well, it's not the weekend yet, but I spent a few hours visiting all the domains in your list, and I categorized them. I'll look into uploading a spreadsheet with the results when I have a few minutes.

_________________
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5


Fri Oct 27, 2017 7:07 am
Profile

Posts: 69
Location: Moscow, Russia
os: bsd
Reply with quote
Post Re: Bit domains statistics
biolizard89 wrote:
and I categorized them

How ? In mind ? Or somehow else ?
biolizard89 wrote:
I'll look into uploading a spreadsheet with the results when I have a few minutes.

ok, I will update it to up-to-date.

_________________
bitname.ru:
- whois service for .bit: whois.bitname.ru or whois.bitname.bit
- dns servers for .bit: dns1.bitname.ru dns2.bitname.ru or dns1.bitname.bit dns2.bitname.bit
- bit domains statistics
github


Sat Oct 28, 2017 5:01 am
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 12 posts ]  Go to page 1, 2  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.