Module Request: Keep a Namespace in Sync

Forum rules
Warning !
Avoid using binary softwares from untrusted users.
Prefer compiling it yourself and verify sources.
phelix
Posts: 1631
Joined: Thu Aug 18, 2011 6:59 am

Module Request: Keep a Namespace in Sync

Post by phelix » Wed Jan 13, 2016 10:42 am

Say I launch my app and do a name_filter or something to get a whole namespace. Then I want to keep it in sync without doing the slow name_filter operation again. Probably this would have to parse blocks some way and update all changed names/values.

It would be nice to have a general (Python) module that implements this.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

cassini
Posts: 332
Joined: Sun May 26, 2013 6:36 pm

Re: Module Request: Keep a Namespace in Sync

Post by cassini » Mon Jan 18, 2016 4:38 pm

name_filter is a nice tool for a quick lookup from the command line, but it is not really optimized for precision namespace queries. I remember some issues when a while ago I wrote a script that counts the members of each namespace. I first tried name_filter, and I quickly bumped into all sorts of cases that required special treatment.
That's why I think the module you suggested should avoid name_filter altogether. Instead, it should sift through the 600000+ existing names and do the filtering without relying on name_filter. This may take a few minutes, though.

biolizard89
Posts: 1971
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Module Request: Keep a Namespace in Sync

Post by biolizard89 » Sun Jan 24, 2016 1:22 pm

cassini wrote:name_filter is a nice tool for a quick lookup from the command line, but it is not really optimized for precision namespace queries. I remember some issues when a while ago I wrote a script that counts the members of each namespace. I first tried name_filter, and I quickly bumped into all sorts of cases that required special treatment.
That's why I think the module you suggested should avoid name_filter altogether. Instead, it should sift through the 600000+ existing names and do the filtering without relying on name_filter. This may take a few minutes, though.
Last I heard, as part of the UNO commitments code, the ordering of names was changed to be lexicographic rather than length-based. Once this is merged, name_filter will be a lot less necessary.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

phelix
Posts: 1631
Joined: Thu Aug 18, 2011 6:59 am

Re: Module Request: Keep a Namespace in Sync

Post by phelix » Tue Jan 26, 2016 10:50 am

cassini wrote:name_filter is a nice tool for a quick lookup from the command line, but it is not really optimized for precision namespace queries. I remember some issues when a while ago I wrote a script that counts the members of each namespace. I first tried name_filter, and I quickly bumped into all sorts of cases that required special treatment.
That's why I think the module you suggested should avoid name_filter altogether. Instead, it should sift through the 600000+ existing names and do the filtering without relying on name_filter. This may take a few minutes, though.
Retrieving (all?) ~674k names with name_scan takes me about 50 seconds. (I wonder whether namehistory=1 makes a difference for name_scan - I have it activated). BTW getting name_scan right seems quite difficult to me (depending on the nb argument and proper escaping of the start name?).

But what I am after is continually updating the dataset with incoming blocks without having to dig through it all again.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

cassini
Posts: 332
Joined: Sun May 26, 2013 6:36 pm

Re: Module Request: Keep a Namespace in Sync

Post by cassini » Tue Jan 26, 2016 11:20 am

phelix wrote:BTW getting name_scan right seems quite difficult to me
The issue I had encountered with using name_scan was a timeout problem. As soon as I had requested more than 250000 entries I got a timeout error (I haven't tried with the latest 0.12 Namecoin Core, though). The solution is to make use of the "start" and "count" parameters of name_scan. (I chose "count 10000" as this is quite convenient for debugging.)

biolizard89
Posts: 1971
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Module Request: Keep a Namespace in Sync

Post by biolizard89 » Tue Jan 26, 2016 2:46 pm

Bitcoin Core has a nice notification API. See https://github.com/namecoin/namecoin-core/issues/33 for the discussion of adding name notifications. I'm not sure why this thread was even created, given that Phelix had already said in the GitHub discussion that the feature he just requested in this thread wasn't needed....
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

phelix
Posts: 1631
Joined: Thu Aug 18, 2011 6:59 am

Re: Module Request: Keep a Namespace in Sync

Post by phelix » Tue Jan 26, 2016 3:33 pm

biolizard89 wrote:Bitcoin Core has a nice notification API. See https://github.com/namecoin/namecoin-core/issues/33 for the discussion of adding name notifications. I'm not sure why this thread was even created, given that Phelix had already said in the GitHub discussion that the feature he just requested in this thread wasn't needed....
The use case is different. This is about applications that need to work on a whole namespace, e.g. to do reverse lookups.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

biolizard89
Posts: 1971
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Module Request: Keep a Namespace in Sync

Post by biolizard89 » Tue Jan 26, 2016 5:09 pm

phelix wrote:
biolizard89 wrote:Bitcoin Core has a nice notification API. See https://github.com/namecoin/namecoin-core/issues/33 for the discussion of adding name notifications. I'm not sure why this thread was even created, given that Phelix had already said in the GitHub discussion that the feature he just requested in this thread wasn't needed....
The use case is different. This is about applications that need to work on a whole namespace, e.g. to do reverse lookups.
First off, it is not at all clear to me why reverse lookups would need to read a whole namespace. Do you expect that approach to scale, at all?

More on topic, how is this a different use case to what's in that GitHub issue? The issue was on getting notifications from namecoind when name transactions appear. The filtering method is explicitly not fixed in the description in that issue.
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

phelix
Posts: 1631
Joined: Thu Aug 18, 2011 6:59 am

Re: Module Request: Keep a Namespace in Sync

Post by phelix » Wed Jan 27, 2016 10:37 am

what I read: :D
Maybe domob's suggestion we discussed a while ago might be helpful to acomplish this.
Yeah, thanks for reminding me. It might help but I think it should be possible to accomplish this cleanly outside the lean and mean core client. I thought of parsing incoming blocks - do you think that could work?
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

biolizard89
Posts: 1971
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Module Request: Keep a Namespace in Sync

Post by biolizard89 » Wed Jan 27, 2016 8:55 pm

To be honest, I don't think the decision of whether this should be part of Core is really our decision. Bitcoin Core devs have a notification API which is pretty advanced (although I haven't used it myself) and well-maintained. That API seems to be directly applicable to the problem brought up in this thread. So I think there's a strong case to be made that reusing the Core notification API makes sense, because why else would that API even exist?

Parsing blocks is what Armory and mSIGNA do. Armory reads blocks from the filesystem, while mSIGNA reads blocks using the wire protocol. This *kind of* makes sense for what they're doing, i.e. implementing a full client that only relies on Core for verifying consensus rules. Our use case doesn't really fall there, I think. We could, hypothetically, parse blocks using RPC getblock followed by lots of gettransaction... but it is not at all clear to me why that would have any advantages over using a notification API. Using RPC to query this stuff works great when you're doing random access. Accessing all wallet or namespace specific transactions, at the exact time that they enter the mempool or blockchain, is anything but random access, and there are much better ways to address that use case.

Also worth pointing out that parsing blocks won't give you unconfirmed name operations, despite the fact that those are almost always safe (and in some cases, security critical, e.g. when someone revokes a TLS key).
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

Post Reply