Berkeley DB

[indolering]

UTXO is under active development.

Are you not planning on porting over Huntercoin to Libcoin? I'm pretty ignorant about the detail here, but I know that Micheal G. of Libcoin is planning on removing BDB entirely. Although, I'm guessing you already knew this : )

[domob]

UXTO is under active development.

Are you not planning on porting over Huntercoin to Libcoin? I'm pretty ignorant about the detail here, but I know that Micheal G. of Libcoin is planning on removing BDB entirely. Although, I'm guessing you already knew this : )

We were planning this, but I'm not really sure any more. For me, running Namecoin with libcoin (although in persistence=full mode, so not using his UTXO implementation) uses much more disk space and is even more painful to sync than with namecoind. I get the impression that BDB is indeed more efficient for storage than his SQLite database. It is just my personal impression, though ... it would be good to get other opinions.

I think it is worthwhile to add the missing Namecoin RPC commands in any case, in order to get at least an alternative client. But porting over Huntercoin would be much more work, since we would have to port also the game engine and other protocol extensions necessary. It is not the current priority, at least.

[indolering]

UXTO is under active development.

Are you not planning on porting over Huntercoin to Libcoin? I'm pretty ignorant about the detail here, but I know that Micheal G. of Libcoin is planning on removing BDB entirely. Although, I'm guessing you already knew this : )

We were planning this, but I'm not really sure any more. For me, running Namecoin with libcoin (although in persistence=full mode, so not using his UTXO implementation) uses much more disk space and is even more painful to sync than with namecoind. I get the impression that BDB is indeed more efficient for storage than his SQLite database. It is just my personal impression, though ... it would be good to get other opinions.

I think it is worthwhile to add the missing Namecoin RPC commands in any case, in order to get at least an alternative client. But porting over Huntercoin would be much more work, since we would have to port also the game engine and other protocol extensions necessary. It is not the current priority, at least.

Is the Bitcoin codebase so divergent that we could not switch using LevelDB?

We could ask Micheal to weigh in, I had an email exchange with him about the use of BDB. For an exchange, well, the creation of relational databases were largely driven by banks needing to balance the books! There are adapters for SQL queries for both LevelDB and BDB. Of course, my experience in this area is skin deep.

What I really worried about is how servicing lite-clients using UTXO, SCIP, and SPV impact the mainline architecture. I used CouchDB because it lets me preprocess their queries, but the security model relies on public key crypto and the choice of CouchDB was based on clients having flaky storage and data connections. I'm guessing that the CouchDB model will be put into maintenance mode within a year or two. But I should just dive into the code if I want an answer to that question....

[phelix]

Were there any benefits from switching to a newer Berkeley DB version? I heard it would be relatively easy but wallets were not compatible.

[domob]

BTW: https://stackoverflow.com/questions/234 ... e/23475187

For me, shutting down Namecoin (and in particular also Huntercoin) take a very long time. Most of this time seems to be spent in the DbEnv::lsn_reset call during DBFlush(). According to the answer to my question on StackOverflow, this call completely rewrites the database. (Not sure why exactly that would be necessary, but it seems to be like that.) On the other hand, if we don't use lsn_reset, it seems we need to keep the DB log files and are also not able to securely move the blockchain files form one machine to another.

Should we try to make it configurable whether or not lsn_reset should be called to make the DB portable and remove the logs? Or maybe only call it from time to time when the logs grow too large, but not always? I think that it is a very bad idea to completely rewrite all databases (except the blk000?.dat files which are not in BDB) on each shut down - I really don't understand why BDB needs to do that.

[phelix]

BTW: https://stackoverflow.com/questions/234 ... e/23475187

For me, shutting down Namecoin (and in particular also Huntercoin) take a very long time. Most of this time seems to be spent in the DbEnv::lsn_reset call during DBFlush(). According to the answer to my question on StackOverflow, this call completely rewrites the database. (Not sure why exactly that would be necessary, but it seems to be like that.) On the other hand, if we don't use lsn_reset, it seems we need to keep the DB log files and are also not able to securely move the blockchain files form one machine to another.

Should we try to make it configurable whether or not lsn_reset should be called to make the DB portable and remove the logs? Or maybe only call it from time to time when the logs grow too large, but not always? I think that it is a very bad idea to completely rewrite all databases (except the blk000?.dat files which are not in BDB) on each shut down - I really don't understand why BDB needs to do that.

I don't see a particular reason for the reset on shutdown (assuming the wallet is always flushed!). But the DB should be cleaned up at some point.

[domob]

BTW: https://stackoverflow.com/questions/234 ... e/23475187

For me, shutting down Namecoin (and in particular also Huntercoin) take a very long time. Most of this time seems to be spent in the DbEnv::lsn_reset call during DBFlush(). According to the answer to my question on StackOverflow, this call completely rewrites the database. (Not sure why exactly that would be necessary, but it seems to be like that.) On the other hand, if we don't use lsn_reset, it seems we need to keep the DB log files and are also not able to securely move the blockchain files form one machine to another.

Should we try to make it configurable whether or not lsn_reset should be called to make the DB portable and remove the logs? Or maybe only call it from time to time when the logs grow too large, but not always? I think that it is a very bad idea to completely rewrite all databases (except the blk000?.dat files which are not in BDB) on each shut down - I really don't understand why BDB needs to do that.

I don't see a particular reason for the reset on shutdown (assuming the wallet is always flushed!). But the DB should be cleaned up at some point.

We could implement a more flexible approach in DBFlush(), that, for instance, calls lsn_reset for a full flush for the wallet but not the blkindex or nameindex. For those, we could only flush from time to time if the log files become "too large" in some sense. I'll have to investigate how the DB behaves without the lsn_reset to find out how to gauge the parameters and how it will do with these changes applied.