[NMDF] Namecoin API Server [0.5BTC] [frozen]

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by phelix »

indolering wrote:
phelix wrote: Sure thing. Sounds great, looking forward to this. Hope it will be integrated with the Firefox plugin someday so that we will have lightweight Namecoin Tor with TLS. :mrgreen:
It's funny that you posted this, I just got the blockchain dumped into CouchDB and I'm hoping to have the client-side front end working tomorrow.
Nice. Could you explain the system structure? Custom Python script & CouchDB?
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

indolering
Posts: 801
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by indolering »

Node.js uses RPC calls to namecoind name_filter in batches of 10K. It tries to encode each value as JSON, changing the "reserved" values into {'$reserved':value} and tossing any key/value pair if JSON.parse throws any errors. Then it pushes all 10K into CouchDB. I spent the day trying to make the import more reliable.

Now I mainly need to work on incremental updates, saving any key/value pairs which CouchDB claims already exist and grabbing each individually from the blockchain and overwriting whatever exists in CouchDB. I'm also having trouble getting CORS to work, so the client cant just call the DB directly. I probably won't have time to fix these issues until this weekend, however.

Most of the blockchain + filtering out names with empty vars:

Code: Select all

https://indolering.cloudant.com/namecoin/_all_docs?limit=20
https://indolering.cloudant.com/namecoin/name
Filtering out obviously useless values:

Code: Select all

https://indolering.cloudant.com/namecoin/_design/meta/_view/active?limit=20&reduce=false

https://indolering.cloudant.com/namecoin/_design/meta/_view/active
You can yank off the limit parameter on the active doc as well, it loads instantly.
DNS is much more than a key->value datastore.

indolering
Posts: 801
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by indolering »

Alright, I'm calling this one as being done: I've got a namecoind daemon running on a VPS, a scraping script which transfers new names to a local CouchDB instance, and a replication feed to frontend CouchDB servers at Cloudant.

As stated before, CouchDB comes with batteries included: it provides a DB, a server, and it has a full-featured rest API. You can read the full API here or start pulling queries using the URL below (it auto-detects content type for the headers).

Code: Select all

https://indolering.cloudant.com/bit/name
CouchDB was architected for read-heavy use and its http protocol and replication model enable really, really great scaling, so it should be fairly resilient to sudden surges in traffic and attacks on the service.

If you want to start using it in a webapp setup a CouchDB instance and turn on continuous replication of the https://indolering.cloudant.com/bit database. Alternatively you can setup namecoind and CouchDB instances on your own server, git clone node-namecoin, and setup a cron job (this is not for the faint of heart).

The service is definitely in "beta" for now, I'm having trouble getting a valid SSL connection over to Cloudant. Do not trust this data feed until April when I get a chance to redo the server using Ubuntu 14, I couldn't get Namecoin to build on Ubuntu 12 so I am using an unfamiliar distro and outdated versions of software. Eventually, I would like to sign all API calls eventually so we don't have to trust the CDN's not to fiddle with the data.

Phelix, I would like you to defer dispensing funds for now. Bounty Source's Bitcoin feature is only being held up by a problem on Coinbase's end and and a .5 BTC payout would be a great way to start. I'm planning on recycling the funds –setting up more bounties for the record signing and rewriting the scrapper– and I would like our first payout using BountySource to set that example. I would also like to help out Sugarpuff as he pushed his out a few days ago. I don't really understand what he is doing and I called "dibs" on this bounty "first" but I think that's pretty childish, he also seems to have a more secure (but not as scalable nor as usable) API and we will probably work together in the future on it.
DNS is much more than a key->value datastore.

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by phelix »

indolering wrote:Alright, I'm calling this one as being done: I've got a namecoind daemon running on a VPS, a scraping script which transfers new names to a local CouchDB instance, and a replication feed to frontend CouchDB servers at Cloudant.

As stated before, CouchDB comes with batteries included: it provides a DB, a server, and it has a full-featured rest API. You can read the full API here or start pulling queries using the URL below (it auto-detects content type for the headers).

Code: Select all

https://indolering.cloudant.com/bit/name
CouchDB was architected for read-heavy use and its http protocol and replication model enable really, really great scaling, so it should be fairly resilient to sudden surges in traffic and attacks on the service.

If you want to start using it in a webapp setup a CouchDB instance and turn on continuous replication of the https://indolering.cloudant.com/bit database. Alternatively you can setup namecoind and CouchDB instances on your own server, git clone node-namecoin, and setup a cron job (this is not for the faint of heart).

The service is definitely in "beta" for now, I'm having trouble getting a valid SSL connection over to Cloudant. Do not trust this data feed until April when I get a chance to redo the server using Ubuntu 14, I couldn't get Namecoin to build on Ubuntu 12 so I am using an unfamiliar distro and outdated versions of software. Eventually, I would like to sign all API calls eventually so we don't have to trust the CDN's not to fiddle with the data.

Phelix, I would like you to defer dispensing funds for now. Bounty Source's Bitcoin feature is only being held up by a problem on Coinbase's end and and a .5 BTC payout would be a great way to start. I'm planning on recycling the funds –setting up more bounties for the record signing and rewriting the scrapper– and I would like our first payout using BountySource to set that example. I would also like to help out Sugarpuff as he pushed his out a few days ago. I don't really understand what he is doing and I called "dibs" on this bounty "first" but I think that's pretty childish, he also seems to have a more secure (but not as scalable nor as usable) API and we will probably work together in the future on it.
Wooha it works. It's completely different than what I asked for but still. Give some time to look at / think about / discuss.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

indolering
Posts: 801
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by indolering »

No problem, I'm only worried about covering ongoing server costs : )

It is quite different on the backend, but I think that's more an artifact of choosing the right solution for the problem. The entire architecture is designed around REST APIs. You can twist Python into doing that, but why?

Anyway, I think we should break up the remainder into smaller goals like getting id/ working, setting up a change subscriptions from LevelDB, and record signing. Eventually, we will need to invest in physical signing equipment and NSL-proof hosting.
DNS is much more than a key->value datastore.

sugarpuff
Posts: 110
Joined: Tue Oct 22, 2013 10:17 pm

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by sugarpuff »

Hey, so uh... isn't this a request for DNSChain? Do I get the 0.5 BTC? :P

It's CoffeeScript though, not python.

Edit: I see indolering mentioned my name in his reply.. I'd be happy to work with him but I believe DNSChain already does most (or all) of what was asked..? Please have a look at the DNSChain page though, the approaches are rather different. One is new software (and does more than requested in the original post), the other combines namecoind/nmcontrol with CouchDB and doesn't introduce any new functionality beyond what that combination offers.
Last edited by sugarpuff on Thu Feb 20, 2014 5:03 am, edited 2 times in total.

sugarpuff
Posts: 110
Joined: Tue Oct 22, 2013 10:17 pm

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by sugarpuff »

FYI, I've exposed my personal DNSChain server here: dns.dnschain.net. You can query it:

- http://dns.dnschain.net/id/greg
- http://dns.dnschain.net/d/dot-bit

sugarpuff
Posts: 110
Joined: Tue Oct 22, 2013 10:17 pm

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by sugarpuff »

I'll go through the list, with the answers for DNSChain:
phelix wrote:This would allow for lightweight applications: Firefox browser plugin with TLS, file signatures, name browsing

* Should use Nmcontrol (integrated or query) or maybe a custom Python script
Uses JSON-RPC2 to talk to namecoind, no need for nmcontrol.
* Server should run on nginx, tornado, supervisord
I have nginx proxying requests to it, and DNSChain, like nginx, is an asynchronous, high-performance server written on top of NodeJS.
See some old config files of mine here: http://blockchained.com/stuff/tornado_config.7z (includes Namecoin TLS support)
Don't think this is necessary...?
* Should implement some kind of query time / size limit and DDOS protection in Nmcontrol
Don't use NMControl, and this is easily done using nginx's rate limiting options.
* Should have an ICANN domain name or subdomain (not on namecoin.org)
dns.dnschain.net (as examples above show), but that's just my personal server I'm letting others use for free.
* Has to run for at least one year
Has been holding steady without any problems since Feb 7th, I think.
* All files need to be open source on github with a free license
Check.
* Should come with an example query file in python
Not sure what you need this for... or why it needs to be in python? It's just HTTP (language agnostic).
* a .bit domain with TLS support would be nice (should be very simple with the config files above)
DNSChain is a DNS server, and so you can load .bit domains over it. To make HTTPS warnings go away, something like a browser extension is required.
* should be reasonably performant (cached name list)
It is. Caching can be done in many ways (PowerDNS, built-in memory, using memcached, etc.).
* For much later: offer some security by adding the original block + a couple of following blockheaders
Not sure what you mean by this, but DNSChain will soon sign all its replies.
API (json):
* name_show (just like namecoind or nmcontrol)
* name_show at previous height, return failure if older than e.g. 100k blocks (nice to have)
Just load the URLs, it's a RESTful API (examples given already).
Edit:
You will receive 0.3BTC now and 0.2BTC after one year.
Query names could be hashed and results encrypted with the name or something to protect the server operator from liabilities.

edited 2014-01-19
DNSChain supports dnscrypt! See the GitHub page (I'm using it now!). :)

As for encrypting HTTPS, that can also be added easily using NodeJS (and will be added soon). Currently HTTP is supported.

indolering
Posts: 801
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by indolering »

Update, oops, I saw Sug posted before I got this up. I've got pizza to eat, but I'm not sure there is much to respond to : )
sugarpuff wrote: Edit: I see indolering mentioned my name in his reply.. I'd be happy to work with him. Please have a look at the DNSChain page though, the approaches are rather different. One is new software (and does more than requested in the original post), the other combines namecoind/nmcontrol with CouchDB and doesn't introduce any new functionality beyond what that combination offers.
The scraper on the next version will definitely NOT be in Javascript. Libcoin embeds SQLite so I should be able to just include triggers to POST changes directly to CouchDB. Eventually, I would like to embed the replication protocol directly in Libcoin so it can sync with CouchDB directly.

I should point out that we are in no way tied to CouchDB any more than any other database we choose. But CouchDB is designed around Map/Reduce and read-heavy operations and it does 99% of the work for us and application developers. It's NOT so great if you want a general database or super-low latency applications, but we aren't going for that. Full-nodes are a good thing and it extends Namecoin more easily to the browser and P2P networking.
me, from another post wrote:For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.

Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow :)
Besides, after UXTO arrives (~year, maybe two) the backend debate goes away, it's essentially becomes a legacy tech.
DNS is much more than a key->value datastore.

sugarpuff
Posts: 110
Joined: Tue Oct 22, 2013 10:17 pm

Re: [NMDF] Namecoin API Server [0.5BTC]

Post by sugarpuff »

indolering wrote:Besides, after UXTO arrives (~year, maybe two) the backend debate goes away, it's essentially becomes a legacy tech.
So, you told me this via IRC and I didn't understand what you meant by it there. Likewise, it makes no sense to me here.

What brief reading I did on UXTO tells me it doesn't supersede the functionality that DNSChain gives.

Locked