Does Google index content on .bit TLD?

[markajansen]

Hi, I am wondering whether Google indexes content on a .bit TLD?

[khal]

No, google doesn't know how to resolve .bit domains, so, unless they explicitly query a .bit dns server or build their own, they won't know .bit.

But, they can discover proxies. For example :
https://encrypted.google.com/search?q=- ... amecoin.us

[nodemaster]

Just wanted to let you know, that I'm already crawling dot-bit websites for some time. If you want to search on .bit TLD please testdrive http://gee.bit/. It's still crawling. It's still alpha and I'm in the process of getting comfortable with mnogosearch (never tried it on that much sites). But it is already functional. I'd be glad if you give some feedback.

[markajansen]

thanks!

[phelix]

Just wanted to let you know, that I'm already crawling dot-bit websites for some time. If you want to search on .bit TLD please testdrive http://gee.bit/. It's still crawling. It's still alpha and I'm in the process of getting comfortable with mnogosearch (never tried it on that much sites). But it is already functional. I'd be glad if you give some feedback.

I see bitcoinX is all crawled. nice.

[khal]

I've added a link to the search engine on the tools page : http://dot-bit.org/tools/

[lehmansister]

The best way to do this, if you actually want google indexing the content, would probably be to have the current crawler output OAI Data (http://www.openarchives.org/). However, I don't really recommend giving it away for free (why would you do free digital work for them?). Maybe let their spiders just get some of the basic index, but leave them with good summary teaser content data not real content.

They are pulling tarball files currently, so are begining to do 'deep indexing.' They pulled about 300Gb of useless auto-generated data from my site before their bot gave up.

Full OAI searching would be nice to see however, but I'd probably just run my own server at home.