[RPC] Remove auxpow from the block index

[domob]

I've noticed that Huntercoin (and because of that probably also Namecoin) spends a lot of its time during start-up in reading the block index data structure (blkindex.dat file). Looking at the code, it seems that this file contains not only basic block header stuff (like previous block hash, merkle root and all that) but also the auxpow (for merged mining). I think I read once that this can be quite large, around 10 KiB or so per block.

In my opinion, the auxpow is not really necessary for the block index. This data structure is used, AFAIK, only to keep the basic "shape" of the block chain in memory. In particular, it seems that auxpow of class CBlockIndex is only every really used in CBlockIndex::GetBlockHeader, which in turn is only used for two things:

• CBlock::ReadFromDisk: If the transactions are not necessary, the block header is instead loaded from CBlockIndex instead of actually read from disk. Not sure when/how often this is the case, but in principle we could read the auxpow in this case when it is first needed instead of loading it for all block index blocks at start-up.
• Processing the "getheaders" network message. Presumably, it is necessary to send the auxpow to peers together with the block headers, so that they can verify the PoW and find the longest chain. Again, I'm not sure how common this will really be in practice -- and so, here we could also load the auxpow from the blocks themselves when necessary.

Does this sound reasonable? I think we could try to remove auxpow from the block index (at least not store it on disk). Please let me know if I'm missing something here ... but otherwise I would be interested to implement this. It could help with start-up times and also reduce the disk space significantly (although this is just guesswork for now, I have not yet done any tests).

[phelix]

Sounds reasonable to my superficial understanding

Does litecoin still have a blkindex.dat? Maybe comparing the sizes could give us some insight on what could be saved. Anybody here that can tell us the size of Litecoin's blkindex.dat?

On the other hand side disk size is cheap. Why does the loading take so long, maybe processing of the auxpow could be reduced at initial load of blkindex.dat.

[MWD]

I think that the biggest roadblock to Namecoin adoption is the five to ten minutes the wallet takes to open, and the fact that during that time, slower computers can't do much else for a few minutes of that.

Fixing that should be the number-one goal of the dev team in my opinion.

If you can fix that in any way that makes sense then I believe that Namecoin can finally take off and fly and gain widespread adoption.

This feedback is based on talking to somewhere between 75 and 100 people I've helped one-on-one with installing the wallet, to use MeowBit.


MWD

[virtual_master]

I've noticed that Huntercoin (and because of that probably also Namecoin) spends a lot of its time during start-up in reading the block index data structure (blkindex.dat file). Looking at the code, it seems that this file contains not only basic block header stuff (like previous block hash, merkle root and all that) but also the auxpow (for merged mining). I think I read once that this can be quite large, around 10 KiB or so per block.

In my opinion, the auxpow is not really necessary for the block index. This data structure is used, AFAIK, only to keep the basic "shape" of the block chain in memory. In particular, it seems that auxpow of class CBlockIndex is only every really used in CBlockIndex::GetBlockHeader, which in turn is only used for two things:

• CBlock::ReadFromDisk: If the transactions are not necessary, the block header is instead loaded from CBlockIndex instead of actually read from disk. Not sure when/how often this is the case, but in principle we could read the auxpow in this case when it is first needed instead of loading it for all block index blocks at start-up.
• Processing the "getheaders" network message. Presumably, it is necessary to send the auxpow to peers together with the block headers, so that they can verify the PoW and find the longest chain. Again, I'm not sure how common this will really be in practice -- and so, here we could also load the auxpow from the blocks themselves when necessary.

Does this sound reasonable? I think we could try to remove auxpow from the block index (at least not store it on disk). Please let me know if I'm missing something here ... but otherwise I would be interested to implement this. It could help with start-up times and also reduce the disk space significantly (although this is just guesswork for now, I have not yet done any tests).

The intention is very reasonable and could help Namecoin.
However I didn't studied so much in detail and I would have two questions:
- If auxpow is used for merged mining eliminating it what consequences would have on the merged mining process ?
- Does it require hardfork ?

[domob]

The intention is very reasonable and could help Namecoin.
However I didn't studied so much in detail and I would have two questions:
- If auxpow is used for merged mining eliminating it what consequences would have on the merged mining process ?
- Does it require hardfork ?

No to both. The auxpow will still be stored in the blk000?.dat files (i. e., the blockchain), communicated over the network and verified when validating blocks as usual. The only thing that would change is that we no longer (also) store it in blkindex.dat. This file is read on start-up, because it contains the block headers and they are kept fully in memory. (This could also be reduced in the future, possibly, since old block headers probably don't need to be stored in memory.)

[domob]

https://github.com/namecoin/namecoin/pull/68

Please test and also let me know if it really reduces start-up times for you. On the system I'm currently on, start-up usually only takes a few seconds anyway. But IMHO, this patch should reduce the time for loading the blockchain initially from disk drastically.

Note that blkindex.dat also contains an index of all transactions ever made, this makes up a large chunk of its size. I'm not sure if it is strictly necessary (keeping only unspent outputs in the index would be enough in principle), but it can't be removed for now (because it could break RPC calls like getrawtransaction). It shouldn't hurt start-up performance, though, since only the blkindex entries are loaded during start-up.

[domob]

First result on my (slower) laptop with mechanical hard disk (after a complete restart, so nothing cached in memory yet): 733 seconds in "Loading block index..." without the patch, 466 seconds with it (and after the first start that updates the database format). I'll also test what the timing is after "compressing" the database file by dump & load, maybe it gets even better.

But again: Please test that everything still works. I think it should, but especially for edge cases like reorgs or things that trigger GetBlockHeader, I've not yet done that. I'll run the client with this patch a bit, though.

[phelix]

Benchmarks (SSD, 8GB RAM, several runs, cached):

Old: 15,5 sec
Initial: ~ 40 sec
New: 13 sec

For comparison:
Bitcoin (cold): 27
Bitcoin (hot): 19

Size of blkindex stayed the same. I renamed it and somehow the client failed to rebuild it the first time. On quitting it crashed:

blkindex.dat: unable to flush: No such file or directory
txn_checkpoint: failed to flush the buffer cache: No such file or directory
Database handles still open at environment close
Open database handle: nameindex.dat/main

Started the second time it rebuilt blkindex.dat and it looked like it would start redownloading blocks from the start - argh - but then it froze at 100% one cpu core. blkindex.dat is growing slowly, it is now at 184kb. Kicked it after a couple of minutes.

With this wallet I had once registered a hundred or so names playing with Namecoin stock control. The debug file was at 45mb already because of all the ismine() output. Recompiling with the two ismine() printfs removed (see my latest commit).

On restarting it's displaying the warning about "displayed transactions may not be correct" and it looks like it is resyncing from the start. It's not greyed out or anything but in the debug window blocks are not counting up either....

Restarting with the normal client version and original blkindex.dat: not downloading blocks... maybe nameindex.dat is messed up.

So there still are some issues with this change.

IMHO there are fileio/database problems that slow the client down more than auxpow (the same that haunted Bitcoin for a long time). Could be repeated flushing of log files or something like that.

[domob]

Is deleting blkindex.dat supposed to work? That always failed to recreate properly for me. Does it work without the patch? Note that nameindex and blkindex should be synchronised in theory, but this is not done properly in the client. It doesn't cause problems usually, but may if you delete blkindex or so.

Recreating blkindex.dat with a smaller size is still a problem - you can do it with the BDB command-line tools (db5.1_dump and db5.1_load on my system). I tried using the Db::compact call, which should do that, but it fails claiming it can not allocate enough memory. Not sure what the problem there is.

I can implement a routine to manually rewrite blkindex.dat like what is done with the wallet on encryption.

[phelix]

tbh I am not sure whether the gains justify the risk of this change at the moment. For me the sluggish GUI is much more bothersome than waiting a couple of seconds longer on startup or wasting a couple of MBs.