reading the blockchain

[tuseroni]

i'm working on some software to get the name/value pairs from the blocklist based on certain criteria, thing is..i have no idea how to get it or how to read it. i have been looking over the code but i can't find where this happens. perhaps someone could help point me in the right direction?

[moa]

this topic probably should have gone in "Technical Support" ... I think you can use a few different RPC calls

Code: Select all

namecoind help

specifically

Code: Select all

namecoind help name_filter

and

Code: Select all

namecoind help name_scan

and pipe the output to a file ... or wherever you want to do with it ... can't be any more specific than your question though realistically.

[tuseroni]

not quite what i meant.
i did eventually find that it creates the database "nameindexfull.dat" which from what i get is a berkeley db, so i should be able to read it easy enough, but how does it get this file. i understand it's peer to peer but what protocol does the P2P network use?

[phelix]

Bitcoin. Careful this rabbit hole is deep.

[tuseroni]

so far as i have found from the source code, a message on the p2p network follows this format:
|____| header
|____________| message
|____| length
|____| checksum
where the header is 0xFA,0xBF,0xB5,0xDA(or 0xf9,0xbe,0xb4,0xd9 i've seen both...i'm guessing the first is bitcoin the other is namecoin) the list of commands which are valid i found in main.cpp
i see a message called "getData" is this the command which gets the blockchain? if so does it get the entire chain or just one record (it seems to be in a for loop so i'm thinking just one record, but that wouldn't make much sense because it doesn't specify an index so iterative calls would just grab the same data...)
is there any place that specifies the protocol used?
if not, i might just need to write one with what i've discovered.

[moa]

so far as i have found from the source code, a message on the p2p network follows this format:
|____| header
|____________| message
|____| length
|____| checksum
where the header is 0xFA,0xBF,0xB5,0xDA(or 0xf9,0xbe,0xb4,0xd9 i've seen both...i'm guessing the first is bitcoin the other is namecoin) the list of commands which are valid i found in main.cpp
i see a message called "getData" is this the command which gets the blockchain? if so does it get the entire chain or just one record (it seems to be in a for loop so i'm thinking just one record, but that wouldn't make much sense because it doesn't specify an index so iterative calls would just grab the same data...)
is there any place that specifies the protocol used?
if not, i might just need to write one with what i've discovered.

please do.

[phelix]

https://en.bitcoin.it/wiki/Protocol_specification

[tuseroni]

phelix: thank you, that's exactly what i needed. now i get what it does:
to connect to the network it first gets your external IP from IRC (or failing that from "whatsmyip.org" or another such site) then it get's the peers from "bitseed.xf2.org","bitseed.bitcoin.org.uk", or "dnsseed.bluematt.me" (in that order i think) which means namecoin is dependent on the centralized DNS. following that it's down to the communication protocol. so next it get's the blockchain(using getblocks i think), then it scans the blockchain for name transactions to create the nameindexfull.dat file for quicker name lookups.

which...may be a problem. it's not expected for anyone to have the full historical block chain, and names can be renewed without changing the mapping information for that block. so doesn't this mean after a while the mapping information could get lost to history for names which are still valid and unexpired?

which may be a question for a different topic, in the mean time i think i have everything i need for my program thank you all for your help.

(i almost lost this post. i posted but my login expired, i logged in and came back to this page but it was all gone, so i went forward, and then resent it, it said it was invalid but my text was in the box...yay)

[phelix]

phelix: thank you, that's exactly what i needed. now i get what it does:
to connect to the network it first gets your external IP from IRC (or failing that from "whatsmyip.org" or another such site) then it get's the peers from "bitseed.xf2.org","bitseed.bitcoin.org.uk", or "dnsseed.bluematt.me" (in that order i think) which means namecoin is dependent on the centralized DNS. following that it's down to the communication protocol. so next it get's the blockchain(using getblocks i think), then it scans the blockchain for name transactions to create the nameindexfull.dat file for quicker name lookups.

Are we really still dependent on all that old stuff? Oh boy, we need to rebase on Bitcoin v0.8...

which...may be a problem. it's not expected for anyone to have the full historical block chain, and names can be renewed without changing the mapping information for that block. so doesn't this mean after a while the mapping information could get lost to history for names which are still valid and unexpired?

Where did you get that from?

which may be a question for a different topic, in the mean time i think i have everything i need for my program thank you all for your help.

looking forward to it (while also wondering what it will do)

(i almost lost this post. i posted but my login expired, i logged in and came back to this page but it was all gone, so i went forward, and then resent it, it said it was invalid but my text was in the box...yay)

[/quote]
on windows firefox at least there is lazarus - check it out

[tuseroni]

well, i'm quite confused now. i am connecting to a peer, i send the message, but the peer is ignoring it. i have no idea what is wrong with the message. right not i'm just trying to get to a point where i send a message to the peer, and the peer sends me something back. would someone be willing to look at my horribly messy testing code and divine what i'm doing wrong?
https://github.com/tuseroni/fireDDNS

to phelix:
it seems it does not use those dns, the code to use it is still in there but namecoin.cpp overrides the dns and seed peers (of which there are notably fewer). it also changes the 4th byte of the header to 0xfe (didn't even notice it was doing that til i looked at the message in wireshark) think it still uses the irc, not sure.
i do wonder though why would you need to lookup your address, couldn't the peer tell you your address on first connect? seems like you would just make a connection, leave source ip blank and when the peer responds he will fill in your ip for you (or if it's being routed the first peer to get your connection will fill it in for you) but again, neither here nor there...

as for why it's not expected for anyone to have the whole blockchain, i got that from the bitcoin protocol spec, though i could be misunderstanding:

getdata

getdata is used in response to inv, to retrieve the content of a specific object, and is usually sent after receiving an inv packet, after filtering known elements. It can be used to retrieve transactions, but only if they are in the memory pool or relay set - arbitrary access to transactions in the chain is not allowed to avoid having clients start to depend on nodes having full transaction indexes (which modern nodes do not).

Payload (maximum payload length: 1.8 Megabytes or 50000 entries):

like i said i could be misunderstanding...