Questions for Vinced about merged mining proxy

[shads]

I've spent the last couple of days going through all sorts of mental contortions trying to decipher the backend of merged mining. I'm finally getting a handle on it (in no small part thanks to ArtForz). I'm particularly cranky today because all the diagrams I drew trying to make sense of it I've just put through the washing machine.

I have great reservations about merged-mining-proxy's impact on pool performance and stability and I've been looking at how build it into poolserverj natively.

I also think it's incredibly concerning that it's about to be rolled into production when there's probably less than 5 people who understand how it works. I think we will find the competitive pressure on pools to adopt will cause a far more rapid uptake than many expect. The documentation is adequate from a 'I'll use merged mining the way I'm told' perspective but for anyone to delve deeper requires deciphering the source in two languages (none of which I happen to know). Mike Hearn's overview on the bitcoin wiki explains the principles at an architectural level but there's nothing on the particular implementation that I can find.

Anyway... to the point.

1/ repeating a question I posted here: http://dot-bit.org/forum/viewtopic.php? ... 2058#p2058

It's not clear whether the call to buildmerkletree should include just the aux block hashes or also the parent block hash. If the parent is not included then the merkle tree simply contains the hash of the namecoin block which has been obtained already from getauxblock.

as a side note it would good if the ordering that these should be submitted was documented a little more. It would be very easy to assume the order was by chainid which in practice it is currently but cannot be relied on.

2/ Is there any reason why the merkle tree can't be built locally rather than using an rpc call?

3/ mm-proxy submits every received work to the daemon before checking it against the various difficulties. This will add a non-trivial load to pool servers. As it appears the getworkaux("",data) only returns the hash of the solution and the (merkle_root + chain_size + nonce) this seems redundant. The hash can be calculated locally and for poolserverj and pushpool it already is. The (merkle_root + chain_size + nonce) was previously submitted as a parameter to getworkaux <aux> so it's trivial to keep it on the poolserver side.

Am I missing something here? I note that both the buildmerkletree and the getworkaux("",data) calls can only be avoided if hashing is done locally. Perhaps the lack of a python sha256 function was the reason for doing it remotely?

4/ the most confusing part so far: The PoW return by the call to parent.rpc_getworkaux("", data, chain_merkle_index, *branch). What exactly is in it. I'm assuming there's part of the coinbase transaction in there and the odd merkle branch or two. It seems to vary in length (usually about 500 bytes) and All I can tell is that it contains the merkle root of the chain hashes; and that for each chainId only about 32 bytes are different.

Can you explain what it contains in what order the components are put together (and pulled apart).

5/ is there any particular reason for this rpc dance mediated by a third party proxy? Wouldn't it far more efficient for these functions to internalised into the parent daemon (since it's already needing a patch anyway)?

Despite my concerns I have to say massive props for putting this together. It's a ridiculously complicated set of concepts to get your head around let alone build a working implementation.

[vinced]

I'm going to answer some of these now and some later.

I've spent the last couple of days going through all sorts of mental contortions trying to decipher the backend of merged mining. I'm finally getting a handle on it (in no small part thanks to ArtForz). I'm particularly cranky today because all the diagrams I drew trying to make sense of it I've just put through the washing machine.

I have great reservations about merged-mining-proxy's impact on pool performance and stability and I've been looking at how build it into poolserverj natively.

I also think it's incredibly concerning that it's about to be rolled into production when there's probably less than 5 people who understand how it works. I think we will find the competitive pressure on pools to adopt will cause a far more rapid uptake than many expect. The documentation is adequate from a 'I'll use merged mining the way I'm told' perspective but for anyone to delve deeper requires deciphering the source in two languages (none of which I happen to know). Mike Hearn's overview on the bitcoin wiki explains the principles at an architectural level but there's nothing on the particular implementation that I can find.

Anyway... to the point.

1/ repeating a question I posted here: http://dot-bit.org/forum/viewtopic.php? ... 2058#p2058

It's not clear whether the call to buildmerkletree should include just the aux block hashes or also the parent block hash. If the parent is not included then the merkle tree simply contains the hash of the namecoin block which has been obtained already from getauxblock.

as a side note it would good if the ordering that these should be submitted was documented a little more. It would be very easy to assume the order was by chainid which in practice it is currently but cannot be relied on.

We don't know the parent block hash yet. The parent block is what we are trying to find by mining.

The ordering of chains is by a pseudo-random generator. I've explained why in another post. Since then someone came up with a better solution which I assume other chains will use when they implement merged-mining.

2/ Is there any reason why the merkle tree can't be built locally rather than using an rpc call?

It was just convenient because the C++ code was already written. You can merge that code into the proxy but I wouldn't do it in python.

3/ mm-proxy submits every received work to the daemon before checking it against the various difficulties. This will add a non-trivial load to pool servers. As it appears the getworkaux("",data) only returns the hash of the solution and the (merkle_root + chain_size + nonce) this seems redundant. The hash can be calculated locally and for poolserverj and pushpool it already is. The (merkle_root + chain_size + nonce) was previously submitted as a parameter to getworkaux <aux> so it's trivial to keep it on the poolserver side.

Am I missing something here? I note that both the buildmerkletree and the getworkaux("",data) calls can only be avoided if hashing is done locally. Perhaps the lack of a python sha256 function was the reason for doing it remotely?

Yes, this could also be merged into the proxy.

4/ the most confusing part so far: The PoW return by the call to parent.rpc_getworkaux("", data, chain_merkle_index, *branch). What exactly is in it. I'm assuming there's part of the coinbase transaction in there and the odd merkle branch or two. It seems to vary in length (usually about 500 bytes) and All I can tell is that it contains the merkle root of the chain hashes; and that for each chainId only about 32 bytes are different.

Can you explain what it contains in what order the components are put together (and pulled apart).

This is a CAuxPow suitable for submitting to the aux chain. This can also be merged into the pool software. But be careful with serialization. I'll explain more later.

5/ is there any particular reason for this rpc dance mediated by a third party proxy? Wouldn't it far more efficient for these functions to internalised into the parent daemon (since it's already needing a patch anyway)?

I minimized the changed to bitcoind because I wanted to maximize the chance that the patch would be merged upstream into bitcoin.

Despite my concerns I have to say massive props for putting this together. It's a ridiculously complicated set of concepts to get your head around let alone build a working implementation.

[shads]

I'm going to answer some of these now and some later.
We don't know the parent block hash yet. The parent block is what we are trying to find by mining.

The ordering of chains is by a pseudo-random generator. I've explained why in another post. Since then someone came up with a better solution which I assume other chains will use when they implement merged-mining.

ok so just confirm I should only be including the aux chain hashes.

If I understand correctly you could build the tree out of any number of fake hashes along with the real aux hash and it would still work for the real aux chain since it's only aux that ever tries to verify and it's only verifying that it's own hash is present in the tree?

Meaning that I can hash the parent block header (with a 0 nonce) and include it like I've been doing without breaking anything but it's not necessary. I realize that's completely academic (and pointless) question but it will help confirm that I'm understanding the verification process correctly.

[vinced]

Yes, that is correct.

I thought more about performance.

The pool software should already filter work submitted to the lowest chain difficulty. If that difficulty is greater than 100 almost all the submitted work will be filtered. So submitting work will not load the mm-proxy or bitcoind/namecoind.

That leaves getwork. The only optimization that will really make a difference is to move the getwork/getworkaux to the pool software and have mm-proxy calculate the aux merkle root once a second. Then you will have almost zero load through mm-proxy.

I hope this makes sense.