Merged mining + timetravel fix @19200 - must upgrade

smoothie
Posts: 34
Joined: Sun Jul 10, 2011 4:45 am
os: linux

Re: Merged mining block number

Post by smoothie »

thirdlight wrote:Yes, wait for the code to be ready. Then pick a block ~ 3 weeks away. Should give enough people a chance to get back from holiday & upgrade.

At current rate, that's gonna be about 19500!
Yea I too agree we should make sure that the code is ready and then add the block # in. 19500 still seems reasonable.

If we need more time then I am sure we will give it more time. No rush.

shads
Posts: 26
Joined: Sun Jul 31, 2011 6:36 am
os: linux

Re: Merged mining block number

Post by shads »

nodemaster wrote: My biggest concern ATM is merged-mine-proxy. I'd really like to see the functionality added to pushpoold.

I agree this is a big worry... Once merged starts there's going to a lot more hash power directed at some of the NMC pools. Given that existing NMC pools are likely to be the earliest adopters.

There's already scalability issues for btc pools and from what I can see from the the merged mining proxy code (I don't know python so apologies if I'm wrong) it's not really designed for scale, just as a reference implementation to make it possible to mine without modifying pools/miners.

I'd like to try and integrate this functionality into poolserverj but I'm having trouble getting my head around exactly what needs to be done. Reading the python code isn't making me much wiser... Could someone who understands the requirements possibly give a step-by-step process flow for getwork and submit work?

What I've worked out so far is roughly... getwork:

- build some derivative of merkle-root = aux
- send getworkaux rpc request with aux parameter
- rpc presumably returns work with multiple targets. (somehow keyed to the chain the target is applicable to?)
- build normal work response and set lowest target
- return to client

submit work:

- send getworkaux call to one daemon with solution which returns hash (of what)? + merkle root that corresponds to original work?
- somehow validate if solution solves for any chain by checking the merkle root returned from setworkaux is contained in some other merkle tree we have stored in the proxy.
- for each blockchain (this is the part where I'm really lost): I'm not even going to try an descibe how I'm interpreting this, probably safer to say I'm not...
- for each blockchain that is solved: send getworkaux with solution, merkle index, the entire branch?

nodemaster
Posts: 172
Joined: Wed Jun 15, 2011 12:46 pm
os: linux

Re: Merged mining block number

Post by nodemaster »

shads wrote:
nodemaster wrote: My biggest concern ATM is merged-mine-proxy. I'd really like to see the functionality added to pushpoold.

I agree this is a big worry... Once merged starts there's going to a lot more hash power directed at some of the NMC pools. Given that existing NMC pools are likely to be the earliest adopters.
I'm no longer concerned about this. the daemon runs flawless and has a very small footprint. It's still not the bottleneck. The bottleneck is threading. No implementation uses more than one core.
shads wrote: There's already scalability issues for btc pools and from what I can see from the the merged mining proxy code (I don't know python so apologies if I'm wrong) it's not really designed for scale, just as a reference implementation to make it possible to mine without modifying pools/miners.
Yes, but that's in most cases where people used one server an let all the daemons run on that machine. You have to scale out horizontally. It's a good practice (for several reasons) to have a couple of low powered pushpoold servers (512 MB RAM are more than enough) and load balance them properly.
Access .bit domains with Firefox in 4 easy steps: https://masterpool.eu/proxy
MasterPool Namecoin Mining Pool

shads
Posts: 26
Joined: Sun Jul 31, 2011 6:36 am
os: linux

Re: Merged mining block number

Post by shads »

nodemaster wrote: It's still not the bottleneck. The bottleneck is threading. No implementation uses more than one core.
No implementation of what?
nodemaster wrote:
shads wrote: There's already scalability issues for btc pools and from what I can see from the the merged mining proxy code (I don't know python so apologies if I'm wrong) it's not really designed for scale, just as a reference implementation to make it possible to mine without modifying pools/miners.
Yes, but that's in most cases where people used one server an let all the daemons run on that machine. You have to scale out horizontally. It's a good practice (for several reasons) to have a couple of low powered pushpoold servers (512 MB RAM are more than enough) and load balance them properly.
True but solving scalability problem by throwing more and more hardware at it when you could get multiples of performance on the same hardware isn't ideal... Are you saying that the proxy can keep up with the rate than bitcoind can deliver work? If that's the case are you talking about a bitcoind patched for multithreaded rpc?

nodemaster
Posts: 172
Joined: Wed Jun 15, 2011 12:46 pm
os: linux

Re: Merged mining block number

Post by nodemaster »

shads wrote:
nodemaster wrote: It's still not the bottleneck. The bottleneck is threading. No implementation uses more than one core.
No implementation of what?
No implementation with pushpoold like functionality.
shads wrote:
nodemaster wrote:
shads wrote: There's already scalability issues for btc pools and from what I can see from the the merged mining proxy code (I don't know python so apologies if I'm wrong) it's not really designed for scale, just as a reference implementation to make it possible to mine without modifying pools/miners.
Yes, but that's in most cases where people used one server an let all the daemons run on that machine. You have to scale out horizontally. It's a good practice (for several reasons) to have a couple of low powered pushpoold servers (512 MB RAM are more than enough) and load balance them properly.
True but solving scalability problem by throwing more and more hardware at it when you could get multiples of performance on the same hardware isn't ideal... Are you saying that the proxy can keep up with the rate than bitcoind can deliver work? If that's the case are you talking about a bitcoind patched for multithreaded rpc?
True but consolidating all daemons on one hardware isn't ideal if you also want to achieve high availability and a secure system. If your primary goal however is a cheap system I agree with you.

Don't get me wrong. I was not meant to criticize your hard work and I appreciate that you are adding functionality to your poolserver implementation. I just wanted to let you know that I am no longer concerned about the issue as you were citing me.
Access .bit domains with Firefox in 4 easy steps: https://masterpool.eu/proxy
MasterPool Namecoin Mining Pool

shads
Posts: 26
Joined: Sun Jul 31, 2011 6:36 am
os: linux

Re: Merged mining block number

Post by shads »

I agree re: high availability though I do have to contend the point that no pushpoold like implementation is multithreaded... poolserverj runs every major task group in separate thread pools...

I'm curious what has changed to make you think it's no longer a potential bottleneck? Do you mean it's not a bottleneck for pushpool? If so then that's not necessarily the case for poolserverj. It's several times faster on several metrics at the limit of what's been tested and it's largely because of it's concurrent architecture. My concern is that the merged mining proxy is going to throttle all the advantage that psj has gained. I haven't tried it out but from what I can see from the source it's not designed for highly concurrent throughput...

doublec
Posts: 149
Joined: Mon May 23, 2011 12:47 am
os: linux
Location: Auckland, New Zealand
Contact:

Re: Merged mining block number

Post by doublec »

shads wrote:I agree re: high availability though I do have to contend the point that no pushpoold like implementation is multithreaded... poolserverj runs every major task group in separate thread pools...
I believe namebit uses a multithreaded pushpoold implementation. My pool is also mutlithreaded. Unless they are using a multithreaded namecoind though they are hitting a bottleneck there.

shads
Posts: 26
Joined: Sun Jul 31, 2011 6:36 am
os: linux

Re: Merged mining block number

Post by shads »

BTW if I'm wrong about that and it's not a bottleneck that's great... But if it is I'm not taking a shot at the proxy, It probably does what it does very well, but it would benefit from being embedded into an architecture that was designed for high concurrency, high throughput scenarios.

shads
Posts: 26
Joined: Sun Jul 31, 2011 6:36 am
os: linux

Re: Merged mining block number

Post by shads »

doublec wrote:
shads wrote: Unless they are using a multithreaded namecoind though they are hitting a bottleneck there.
I haven't tried but wouldn't joel katz's multithread rpc patch be able to applied to namecoind? When I switched to a patched bitcoind I got performance increase of several thousand %

vinced
Posts: 63
Joined: Wed May 18, 2011 1:16 am

Re: Merged mining block number

Post by vinced »

shads wrote:...
My concern is that the merged mining proxy is going to throttle all the advantage that psj has gained. I haven't tried it out but from what I can see from the source it's not designed for highly concurrent throughput...
The proxy is written using the python twisted library which does non-blocking I/O, so it actually is designed for high concurrency.

It would be best if a pool operator tested it under load. It's just a matter of doing a getwork in a loop from multiple processes and seeing how many requests per second the system can do. Compare that with getwork directly against bitcoind/namecoind.
!v | Namecoin founder | https://dot-bit.org/

Post Reply