Berkeley DB

[domob]

I get the impression that Berkeley DB is pretty slow and hard on disk accesses. This seems to be the case even though it is usually advertised as industrial strength and very efficient - so maybe we are using it wrong? I still don't understand why it takes more than 700 seconds to load blkindex.dat (without my recent patch), when the file is "only" 500 MiB. I mean, on a modern system, you could load the file entirely into memory and then go through the data there in a fraction of this time...! I'd expect that BDB gives at least almost the same performance as reading the file into memory and then processing it there. We're not doing any complicated look-ups there, just walking over a part of the binary-tree index in order.

Does anyone know anybody who has experience with BDB and can tell us what we are doing wrong here? Maybe we should as a starter increase the default memory cache from 25 MiB (I think that's the value ATM?) to something more reasonable for a modern system?

[phelix]

Yes, more dbcache might help.

What really bothers me is that the client becomes sluggish and with lots of connection (trying with 24 right now, ssd, dbcache=1000) it just greys out for me. I can tell from debug.log that it is downloading blocks but from nothing else.

It should really yield every once in a while and update the GUI.

[domob]

What really bothers me is that the client becomes sluggish and with lots of connection (trying with 24 right now, ssd, dbcache=1000) it just greys out for me. I can tell from debug.log that it is downloading blocks but from nothing else.

It should really yield every once in a while and update the GUI.

In Huntercoin, there was a similar problem: The UI has a timer kicking in every 250 ms that tries to lock some mutexes to update the UI state from the core. While downloading / processing blocks, these locks may wait for a long time, blocking the UI thread for seconds. At least to a lesser extent, that may also be the case with Namecoin-Qt while downloading blocks. For Huntercoin, we replaced the locks with try-locks, so that the UI stays responsive (but may not be 100% up-to-date while downloading blocks).

[phelix]

What really bothers me is that the client becomes sluggish and with lots of connection (trying with 24 right now, ssd, dbcache=1000) it just greys out for me. I can tell from debug.log that it is downloading blocks but from nothing else.

It should really yield every once in a while and update the GUI.

In Huntercoin, there was a similar problem: The UI has a timer kicking in every 250 ms that tries to lock some mutexes to update the UI state from the core. While downloading / processing blocks, these locks may wait for a long time, blocking the UI thread for seconds. At least to a lesser extent, that may also be the case with Namecoin-Qt while downloading blocks. For Huntercoin, we replaced the locks with try-locks, so that the UI stays responsive (but may not be 100% up-to-date while downloading blocks).

That sounds like it could be the reason. I wonder why the gui has to lock any mutexes at all. Can you find the commit?

[domob]

That sounds like it could be the reason. I wonder why the gui has to lock any mutexes at all. Can you find the commit?

Locking is a good idea in general, because otherwise (even if the UI only reads the state) it may display incoherent information.

Looking at the code quickly, the only timer call-back that does lock is NameTableModel::updateExpiration (qt/nametablemodel.cpp:349). It only locks if there are new blocks, but during syncing, that will probably be the case most of the time. Huntercoin uses this code instead of the unconditional lock:

Code: Select all

CTryCriticalBlock criticalBlock(cs_main);
if (!criticalBlock.Entered())
  return false;

I can prepare a corresponding patch for Namecoin if you think this could help. (I'm no user of the Namecoin-UI, though, so I don't really know how bad this is.)

[phelix]

With my stock control testing wallet ("ACME") downloading of the blockchain was so horribly slow I could not believe it. The wallet once had a couple hundred names/transactions so in it.

With the wallet renamed downloading is way faster and the GUI is not sluggish any more.

I wonder whether the GUI can significantly slow down the downloading of blocks? It was my impression. Will give that patch a try.

edit: I had to change the above lines to not return anything. It is quite an improvement. GUI is still sluggish but even with the ACME wallet is does not grey out any more. We should take this into the next (upcoming?) release.

I wonder whether the GUI is requesting expiration information for all names even if the name tab is closed...

[domob]

With my stock control testing wallet ("ACME") downloading of the blockchain was so horribly slow I could not believe it. The wallet once had a couple hundred names/transactions so in it.

With the wallet renamed downloading is way faster and the GUI is not sluggish any more.

I wonder whether the GUI can significantly slow down the downloading of blocks? It was my impression. Will give that patch a try.

edit: I had to change the above lines to not return anything. It is quite an improvement. GUI is still sluggish but even with the ACME wallet is does not grey out any more. We should take this into the next (upcoming?) release.

I wonder whether the GUI is requesting expiration information for all names even if the name tab is closed...

Cool! Do you have a final patch already, or should I prepare one and submit a pull request?

[phelix]

With my stock control testing wallet ("ACME") downloading of the blockchain was so horribly slow I could not believe it. The wallet once had a couple hundred names/transactions so in it.

With the wallet renamed downloading is way faster and the GUI is not sluggish any more.

I wonder whether the GUI can significantly slow down the downloading of blocks? It was my impression. Will give that patch a try.

edit: I had to change the above lines to not return anything. It is quite an improvement. GUI is still sluggish but even with the ACME wallet is does not grey out any more. We should take this into the next (upcoming?) release.

I wonder whether the GUI is requesting expiration information for all names even if the name tab is closed...

Cool! Do you have a final patch already, or should I prepare one and submit a pull request?

I should be able to do it.

There is definitely something going wrong in combination with large wallets:
downloaded & processed blocks in 3 minutes (24 connections)
namecoind: 2500
namecoin-qt (large wallet): 44
namecoind again: 7000
namecoin-qt (fresh wallet): 2251

[snailbrain]

With my stock control testing wallet ("ACME") downloading of the blockchain was so horribly slow I could not believe it. The wallet once had a couple hundred names/transactions so in it.

With the wallet renamed downloading is way faster and the GUI is not sluggish any more.

I wonder whether the GUI can significantly slow down the downloading of blocks? It was my impression. Will give that patch a try.

edit: I had to change the above lines to not return anything. It is quite an improvement. GUI is still sluggish but even with the ACME wallet is does not grey out any more. We should take this into the next (upcoming?) release.

I wonder whether the GUI is requesting expiration information for all names even if the name tab is closed...

Cool! Do you have a final patch already, or should I prepare one and submit a pull request?

I should be able to do it.

There is definitely something going wrong in combination with large wallets:
downloaded & processed blocks in 3 minutes (24 connections)
namecoind: 2500
namecoin-qt (large wallet): 44
namecoind again: 7000
namecoin-qt (fresh wallet): 2251

I apologise if this is not relevant.. but something about incoming transfer of names in the qt is ringing some bells..

[phelix]

I apologise if this is not relevant.. but something about incoming transfer of names in the qt is ringing some bells..

Hmm there should not have been any incoming transfers on that wallet... some ever pending ones, though.