Nice. Could you explain the system structure? Custom Python script & CouchDB?indolering wrote:It's funny that you posted this, I just got the blockchain dumped into CouchDB and I'm hoping to have the client-side front end working tomorrow.phelix wrote: Sure thing. Sounds great, looking forward to this. Hope it will be integrated with the Firefox plugin someday so that we will have lightweight Namecoin Tor with TLS.
[NMDF] Namecoin API Server [0.5BTC] [frozen]
Re: [NMDF] Namecoin API Server [0.5BTC]
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: [NMDF] Namecoin API Server [0.5BTC]
Node.js uses RPC calls to namecoind name_filter in batches of 10K. It tries to encode each value as JSON, changing the "reserved" values into {'$reserved':value} and tossing any key/value pair if JSON.parse throws any errors. Then it pushes all 10K into CouchDB. I spent the day trying to make the import more reliable.
Now I mainly need to work on incremental updates, saving any key/value pairs which CouchDB claims already exist and grabbing each individually from the blockchain and overwriting whatever exists in CouchDB. I'm also having trouble getting CORS to work, so the client cant just call the DB directly. I probably won't have time to fix these issues until this weekend, however.
Most of the blockchain + filtering out names with empty vars:
Filtering out obviously useless values:
You can yank off the limit parameter on the active doc as well, it loads instantly.
Now I mainly need to work on incremental updates, saving any key/value pairs which CouchDB claims already exist and grabbing each individually from the blockchain and overwriting whatever exists in CouchDB. I'm also having trouble getting CORS to work, so the client cant just call the DB directly. I probably won't have time to fix these issues until this weekend, however.
Most of the blockchain + filtering out names with empty vars:
Code: Select all
https://indolering.cloudant.com/namecoin/_all_docs?limit=20
https://indolering.cloudant.com/namecoin/name
Code: Select all
https://indolering.cloudant.com/namecoin/_design/meta/_view/active?limit=20&reduce=false
https://indolering.cloudant.com/namecoin/_design/meta/_view/active
DNS is much more than a key->value datastore.
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: [NMDF] Namecoin API Server [0.5BTC]
Alright, I'm calling this one as being done: I've got a namecoind daemon running on a VPS, a scraping script which transfers new names to a local CouchDB instance, and a replication feed to frontend CouchDB servers at Cloudant.
As stated before, CouchDB comes with batteries included: it provides a DB, a server, and it has a full-featured rest API. You can read the full API here or start pulling queries using the URL below (it auto-detects content type for the headers).
CouchDB was architected for read-heavy use and its http protocol and replication model enable really, really great scaling, so it should be fairly resilient to sudden surges in traffic and attacks on the service.
If you want to start using it in a webapp setup a CouchDB instance and turn on continuous replication of the https://indolering.cloudant.com/bit database. Alternatively you can setup namecoind and CouchDB instances on your own server, git clone node-namecoin, and setup a cron job (this is not for the faint of heart).
The service is definitely in "beta" for now, I'm having trouble getting a valid SSL connection over to Cloudant. Do not trust this data feed until April when I get a chance to redo the server using Ubuntu 14, I couldn't get Namecoin to build on Ubuntu 12 so I am using an unfamiliar distro and outdated versions of software. Eventually, I would like to sign all API calls eventually so we don't have to trust the CDN's not to fiddle with the data.
Phelix, I would like you to defer dispensing funds for now. Bounty Source's Bitcoin feature is only being held up by a problem on Coinbase's end and and a .5 BTC payout would be a great way to start. I'm planning on recycling the funds –setting up more bounties for the record signing and rewriting the scrapper– and I would like our first payout using BountySource to set that example. I would also like to help out Sugarpuff as he pushed his out a few days ago. I don't really understand what he is doing and I called "dibs" on this bounty "first" but I think that's pretty childish, he also seems to have a more secure (but not as scalable nor as usable) API and we will probably work together in the future on it.
As stated before, CouchDB comes with batteries included: it provides a DB, a server, and it has a full-featured rest API. You can read the full API here or start pulling queries using the URL below (it auto-detects content type for the headers).
Code: Select all
https://indolering.cloudant.com/bit/name
If you want to start using it in a webapp setup a CouchDB instance and turn on continuous replication of the https://indolering.cloudant.com/bit database. Alternatively you can setup namecoind and CouchDB instances on your own server, git clone node-namecoin, and setup a cron job (this is not for the faint of heart).
The service is definitely in "beta" for now, I'm having trouble getting a valid SSL connection over to Cloudant. Do not trust this data feed until April when I get a chance to redo the server using Ubuntu 14, I couldn't get Namecoin to build on Ubuntu 12 so I am using an unfamiliar distro and outdated versions of software. Eventually, I would like to sign all API calls eventually so we don't have to trust the CDN's not to fiddle with the data.
Phelix, I would like you to defer dispensing funds for now. Bounty Source's Bitcoin feature is only being held up by a problem on Coinbase's end and and a .5 BTC payout would be a great way to start. I'm planning on recycling the funds –setting up more bounties for the record signing and rewriting the scrapper– and I would like our first payout using BountySource to set that example. I would also like to help out Sugarpuff as he pushed his out a few days ago. I don't really understand what he is doing and I called "dibs" on this bounty "first" but I think that's pretty childish, he also seems to have a more secure (but not as scalable nor as usable) API and we will probably work together in the future on it.
DNS is much more than a key->value datastore.
Re: [NMDF] Namecoin API Server [0.5BTC]
Wooha it works. It's completely different than what I asked for but still. Give some time to look at / think about / discuss.indolering wrote:Alright, I'm calling this one as being done: I've got a namecoind daemon running on a VPS, a scraping script which transfers new names to a local CouchDB instance, and a replication feed to frontend CouchDB servers at Cloudant.
As stated before, CouchDB comes with batteries included: it provides a DB, a server, and it has a full-featured rest API. You can read the full API here or start pulling queries using the URL below (it auto-detects content type for the headers).
CouchDB was architected for read-heavy use and its http protocol and replication model enable really, really great scaling, so it should be fairly resilient to sudden surges in traffic and attacks on the service.Code: Select all
https://indolering.cloudant.com/bit/name
If you want to start using it in a webapp setup a CouchDB instance and turn on continuous replication of the https://indolering.cloudant.com/bit database. Alternatively you can setup namecoind and CouchDB instances on your own server, git clone node-namecoin, and setup a cron job (this is not for the faint of heart).
The service is definitely in "beta" for now, I'm having trouble getting a valid SSL connection over to Cloudant. Do not trust this data feed until April when I get a chance to redo the server using Ubuntu 14, I couldn't get Namecoin to build on Ubuntu 12 so I am using an unfamiliar distro and outdated versions of software. Eventually, I would like to sign all API calls eventually so we don't have to trust the CDN's not to fiddle with the data.
Phelix, I would like you to defer dispensing funds for now. Bounty Source's Bitcoin feature is only being held up by a problem on Coinbase's end and and a .5 BTC payout would be a great way to start. I'm planning on recycling the funds –setting up more bounties for the record signing and rewriting the scrapper– and I would like our first payout using BountySource to set that example. I would also like to help out Sugarpuff as he pushed his out a few days ago. I don't really understand what he is doing and I called "dibs" on this bounty "first" but I think that's pretty childish, he also seems to have a more secure (but not as scalable nor as usable) API and we will probably work together in the future on it.
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: [NMDF] Namecoin API Server [0.5BTC]
No problem, I'm only worried about covering ongoing server costs : )
It is quite different on the backend, but I think that's more an artifact of choosing the right solution for the problem. The entire architecture is designed around REST APIs. You can twist Python into doing that, but why?
Anyway, I think we should break up the remainder into smaller goals like getting id/ working, setting up a change subscriptions from LevelDB, and record signing. Eventually, we will need to invest in physical signing equipment and NSL-proof hosting.
It is quite different on the backend, but I think that's more an artifact of choosing the right solution for the problem. The entire architecture is designed around REST APIs. You can twist Python into doing that, but why?
Anyway, I think we should break up the remainder into smaller goals like getting id/ working, setting up a change subscriptions from LevelDB, and record signing. Eventually, we will need to invest in physical signing equipment and NSL-proof hosting.
DNS is much more than a key->value datastore.
Re: [NMDF] Namecoin API Server [0.5BTC]
Hey, so uh... isn't this a request for DNSChain? Do I get the 0.5 BTC?
It's CoffeeScript though, not python.
Edit: I see indolering mentioned my name in his reply.. I'd be happy to work with him but I believe DNSChain already does most (or all) of what was asked..? Please have a look at the DNSChain page though, the approaches are rather different. One is new software (and does more than requested in the original post), the other combines namecoind/nmcontrol with CouchDB and doesn't introduce any new functionality beyond what that combination offers.
It's CoffeeScript though, not python.
Edit: I see indolering mentioned my name in his reply.. I'd be happy to work with him but I believe DNSChain already does most (or all) of what was asked..? Please have a look at the DNSChain page though, the approaches are rather different. One is new software (and does more than requested in the original post), the other combines namecoind/nmcontrol with CouchDB and doesn't introduce any new functionality beyond what that combination offers.
Last edited by sugarpuff on Thu Feb 20, 2014 5:03 am, edited 2 times in total.
Re: [NMDF] Namecoin API Server [0.5BTC]
FYI, I've exposed my personal DNSChain server here: dns.dnschain.net. You can query it:
- http://dns.dnschain.net/id/greg
- http://dns.dnschain.net/d/dot-bit
- http://dns.dnschain.net/id/greg
- http://dns.dnschain.net/d/dot-bit
Re: [NMDF] Namecoin API Server [0.5BTC]
I'll go through the list, with the answers for DNSChain:
As for encrypting HTTPS, that can also be added easily using NodeJS (and will be added soon). Currently HTTP is supported.
Uses JSON-RPC2 to talk to namecoind, no need for nmcontrol.phelix wrote:This would allow for lightweight applications: Firefox browser plugin with TLS, file signatures, name browsing
* Should use Nmcontrol (integrated or query) or maybe a custom Python script
I have nginx proxying requests to it, and DNSChain, like nginx, is an asynchronous, high-performance server written on top of NodeJS.* Server should run on nginx, tornado, supervisord
Don't think this is necessary...?See some old config files of mine here: http://blockchained.com/stuff/tornado_config.7z (includes Namecoin TLS support)
Don't use NMControl, and this is easily done using nginx's rate limiting options.* Should implement some kind of query time / size limit and DDOS protection in Nmcontrol
dns.dnschain.net (as examples above show), but that's just my personal server I'm letting others use for free.* Should have an ICANN domain name or subdomain (not on namecoin.org)
Has been holding steady without any problems since Feb 7th, I think.* Has to run for at least one year
Check.* All files need to be open source on github with a free license
Not sure what you need this for... or why it needs to be in python? It's just HTTP (language agnostic).* Should come with an example query file in python
DNSChain is a DNS server, and so you can load .bit domains over it. To make HTTPS warnings go away, something like a browser extension is required.* a .bit domain with TLS support would be nice (should be very simple with the config files above)
It is. Caching can be done in many ways (PowerDNS, built-in memory, using memcached, etc.).* should be reasonably performant (cached name list)
Not sure what you mean by this, but DNSChain will soon sign all its replies.* For much later: offer some security by adding the original block + a couple of following blockheaders
Just load the URLs, it's a RESTful API (examples given already).API (json):
* name_show (just like namecoind or nmcontrol)
* name_show at previous height, return failure if older than e.g. 100k blocks (nice to have)
DNSChain supports dnscrypt! See the GitHub page (I'm using it now!).Edit:
You will receive 0.3BTC now and 0.2BTC after one year.
Query names could be hashed and results encrypted with the name or something to protect the server operator from liabilities.
edited 2014-01-19
As for encrypting HTTPS, that can also be added easily using NodeJS (and will be added soon). Currently HTTP is supported.
-
- Posts: 801
- Joined: Sun Aug 18, 2013 8:26 pm
- os: mac
Re: [NMDF] Namecoin API Server [0.5BTC]
Update, oops, I saw Sug posted before I got this up. I've got pizza to eat, but I'm not sure there is much to respond to : )
I should point out that we are in no way tied to CouchDB any more than any other database we choose. But CouchDB is designed around Map/Reduce and read-heavy operations and it does 99% of the work for us and application developers. It's NOT so great if you want a general database or super-low latency applications, but we aren't going for that. Full-nodes are a good thing and it extends Namecoin more easily to the browser and P2P networking.
The scraper on the next version will definitely NOT be in Javascript. Libcoin embeds SQLite so I should be able to just include triggers to POST changes directly to CouchDB. Eventually, I would like to embed the replication protocol directly in Libcoin so it can sync with CouchDB directly.sugarpuff wrote: Edit: I see indolering mentioned my name in his reply.. I'd be happy to work with him. Please have a look at the DNSChain page though, the approaches are rather different. One is new software (and does more than requested in the original post), the other combines namecoind/nmcontrol with CouchDB and doesn't introduce any new functionality beyond what that combination offers.
I should point out that we are in no way tied to CouchDB any more than any other database we choose. But CouchDB is designed around Map/Reduce and read-heavy operations and it does 99% of the work for us and application developers. It's NOT so great if you want a general database or super-low latency applications, but we aren't going for that. Full-nodes are a good thing and it extends Namecoin more easily to the browser and P2P networking.
Besides, after UXTO arrives (~year, maybe two) the backend debate goes away, it's essentially becomes a legacy tech.me, from another post wrote:For shits and giggles, I downloaded the top 1 million domains and some IP blacklists, resulting in a 4MB CSV file containing 142,202 IP <-> domain pairs. Simple Zip compression yielded 2:1 - 3:1 compression rates while PAQ8 delivered >4:1 rate. A custom data-structure would likely yield even higher compression ratios.
Simple extrapolation of these numbers shows that 5megs can easily handle half a million such pairings. If traffic is distributed in a Zipf-like distribution, the .bit namespace can grow to 2.5 million domains (the size the .biz domain space) clients will can store the top 20% of sites which receive 80% of all traffic warmed up in a 5 meg cache. With .INFO carrying 6 million domains, we have room to grow
DNS is much more than a key->value datastore.
Re: [NMDF] Namecoin API Server [0.5BTC]
So, you told me this via IRC and I didn't understand what you meant by it there. Likewise, it makes no sense to me here.indolering wrote:Besides, after UXTO arrives (~year, maybe two) the backend debate goes away, it's essentially becomes a legacy tech.
What brief reading I did on UXTO tells me it doesn't supersede the functionality that DNSChain gives.