[RPC] Remove auxpow from the block index

Namecoin, NMControl
domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: [RPC] Remove auxpow from the block index

Post by domob »

phelix wrote:tbh I am not sure whether the gains justify the risk of this change at the moment. For me the sluggish GUI is much more bothersome than waiting a couple of seconds longer on startup or wasting a couple of MBs.
I don't know about the risks (but in principle there shouldn't be many, I'm running clients with this fine now for a few days), but the start-up is actually very painful for me without a SSD. On your SSD system, you probably don't see much difference - but for me, start-up took more than 10 minutes after a reboot, while with the patch (and compacting the DB manually, TBH) it was only a minute or so.

I forgot to mention this above: After compacting the DB manually (I can try to write a corresponding update to the patch that does this), the reported time of 460 seconds on my system gets reduced to only 10-15 (!). I think this should be worth it.
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

virtual_master
Posts: 541
Joined: Mon May 20, 2013 12:03 pm
Contact:

Re: [RPC] Remove auxpow from the block index

Post by virtual_master »

I also don't use SSD only on Android devices.
With HD the speed depends also of the position of the data on the HD. The last 20% is much slower.
As I have the feeling(without measuring) the loading speed by me:
- on an old 12-13 year old WinXP native on the last 20% of HD partition is about 10-15 min
- on a 1 year old Win8 native AMD quad-core PC 8GB RAM, first 20% of the HD about 1-2 min
- the same PC on Ubuntu 12.04 host/ xubuntu 10.04 guest with 1 core + 1 GB RAM allocated VirtualBox, on the middle of the HD - about 3-4 min

I will look more exactly with the modifications also how will change. Improving the speed is important and will require more steps.
http://namecoinia.org/
Calendars for free to print: 2014 Calendar in JPG | 2014 Calendar in PDF Protect the Environment with Namecoin: 2014 Calendar in JPG | 2014 Calendar in PDF
BTC: 15KXVQv7UGtUoTe5VNWXT1bMz46MXuePba | NMC: NABFA31b3x7CvhKMxcipUqA3TnKsNfCC7S

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: [RPC] Remove auxpow from the block index

Post by phelix »

domob wrote:
phelix wrote:tbh I am not sure whether the gains justify the risk of this change at the moment. For me the sluggish GUI is much more bothersome than waiting a couple of seconds longer on startup or wasting a couple of MBs.
I don't know about the risks (but in principle there shouldn't be many, I'm running clients with this fine now for a few days), but the start-up is actually very painful for me without a SSD. On your SSD system, you probably don't see much difference - but for me, start-up took more than 10 minutes after a reboot, while with the patch (and compacting the DB manually, TBH) it was only a minute or so.

I forgot to mention this above: After compacting the DB manually (I can try to write a corresponding update to the patch that does this), the reported time of 460 seconds on my system gets reduced to only 10-15 (!). I think this should be worth it.
OK, that is quite a difference. 10 Minutes is too much...

I will give it another try without randomly deleting files ;)
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: [RPC] Remove auxpow from the block index

Post by domob »

phelix wrote:OK, that is quite a difference. 10 Minutes is too much...

I will give it another try without randomly deleting files ;)
Thanks! As mentioned already, I can write a proper DB update routine before we finally commit the patch. Also, I will investigate a little further when precisely GetBlockHeader could be called (never happened on my test systems so far) and if we could totally remove this capability (would be more secure against bugs in the new auto-loading of auxpow, if never needed).
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: [RPC] Remove auxpow from the block index

Post by domob »

I've created a fresh pull request: https://github.com/namecoin/namecoin/pull/71 This one is now correctly rebased on the current upstream repository (did I write already that I'm not yet fully fluent with Git?). It also automatically rewrites the blkindex.dat file after the format update, so that it will indeed be fully shrinked without further steps required. The start-up time reduces for me to 17 seconds instead of 700 (!) with this. It should also reduce the memory usage similarly to the size of blkindex.dat.

I've now removed the auxpow field completely from the class CBlockIndex. After some further digging into the code, it was only used for functionality that is never actually called somewhere, as well as from a network command "getheaders" that is never sent by the client. I think removing this dead and possibly untested code is a bonus by itself - it can be re-enabled if it will be used in the future, when it can actually be tested.

I've been running the older version already for some time without problems on different systems I control, and will be testing this one now also. I'll try to resync the blockchain with it.
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: [RPC] Remove auxpow from the block index

Post by phelix »

Will try, hopefully tomorrow.
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

snailbrain
Posts: 309
Joined: Tue Jul 19, 2011 9:33 pm

Re: [RPC] Remove auxpow from the block index

Post by snailbrain »

domob wrote:I've created a fresh pull request: https://github.com/namecoin/namecoin/pull/71 This one is now correctly rebased on the current upstream repository (did I write already that I'm not yet fully fluent with Git?). It also automatically rewrites the blkindex.dat file after the format update, so that it will indeed be fully shrinked without further steps required. The start-up time reduces for me to 17 seconds instead of 700 (!) with this. It should also reduce the memory usage similarly to the size of blkindex.dat.

I've now removed the auxpow field completely from the class CBlockIndex. After some further digging into the code, it was only used for functionality that is never actually called somewhere, as well as from a network command "getheaders" that is never sent by the client. I think removing this dead and possibly untested code is a bonus by itself - it can be re-enabled if it will be used in the future, when it can actually be tested.

I've been running the older version already for some time without problems on different systems I control, and will be testing this one now also. I'll try to resync the blockchain with it.
tested qt on windows

blkindex.dat use 50% ish less space
starts up faster (after first rewrite)
uses 3x less ram (now 90mb, was 270-320mb with version i compared, (3.72)) ??

stuff removed is not needed by miners? read further up

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: [RPC] Remove auxpow from the block index

Post by phelix »

Tested QT on win 8.1 x64, SSD, 8GB RAM, all hot (cached)

Start times:

old: 20s (not sure why it is slower than my previous test when I measured 15.5s; 20-21s over a couple runs was stable now)
size: 573.8mb
ram: 300mb

rewrite: ~128s
size: 306mb

new: 13s
ram: 111mb

Sweet. :mrgreen:

edit:RAM - even better!
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

cointra
Posts: 4
Joined: Wed May 21, 2014 4:48 pm
os: linux

Re: [RPC] Remove auxpow from the block index

Post by cointra »

domob wrote:I've created a fresh pull request: https://github.com/namecoin/namecoin/pull/71 This one is now correctly rebased on the current upstream repository (did I write already that I'm not yet fully fluent with Git?). It also automatically rewrites the blkindex.dat file after the format update, so that it will indeed be fully shrinked without further steps required. The start-up time reduces for me to 17 seconds instead of 700 (!) with this. It should also reduce the memory usage similarly to the size of blkindex.dat.

I've now removed the auxpow field completely from the class CBlockIndex. After some further digging into the code, it was only used for functionality that is never actually called somewhere, as well as from a network command "getheaders" that is never sent by the client. I think removing this dead and possibly untested code is a bonus by itself - it can be re-enabled if it will be used in the future, when it can actually be tested.

I've been running the older version already for some time without problems on different systems I control, and will be testing this one now also. I'll try to resync the blockchain with it.
I see in your pull request that you are removing the getheaders message, I am working on an Android Wallet for Namecoin and it uses getheaders to get headers since it doesn't really maintain complete blockchain. Currently namecoin returns headers with auxpow and so we are parsing the auxpow at the client but that can be removed once your auxpow removal gets integrated in the main client. getheaders seems to work fine during our minimal minimal tests and I would like to suggest that we keep it.

domob
Posts: 1129
Joined: Mon Jun 24, 2013 11:27 am
Contact:

Re: [RPC] Remove auxpow from the block index

Post by domob »

cointra wrote:I see in your pull request that you are removing the getheaders message, I am working on an Android Wallet for Namecoin and it uses getheaders to get headers since it doesn't really maintain complete blockchain. Currently namecoin returns headers with auxpow and so we are parsing the auxpow at the client but that can be removed once your auxpow removal gets integrated in the main client. getheaders seems to work fine during our minimal minimal tests and I would like to suggest that we keep it.
Nice to hear you are working on an Android wallet! Actually, it is not too hard to implement getheaders even with a full header (including the auxpow) - if you need it, we can re-enable it as soon as you will be using it in practice. A previous version of the patch even had the message still supported, but I decided to disable it because I had no way of testing that everything works and I didn't want to leave some untested code in there. But as soon as you have a client to test with and which makes actually use of the feature, I can put it back in.
BTC: 1domobKsPZ5cWk2kXssD8p8ES1qffGUCm | NMC: NCdomobcmcmVdxC5yxMitojQ4tvAtv99pY
BM-GtQnWM3vcdorfqpKXsmfHQ4rVYPG5pKS
Use your Namecoin identity as OpenID: https://nameid.org/

Post Reply