Rebasing namecoin on bitcoin 0.8

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Rebasing namecoin on bitcoin 0.8

Post by khal »

Hi everybody !

As some people should now, I've started to merge namecoin & bitcoin 0.8.

My first attempt was to pick a bitcoin 0.8 and include changes done between namecoin_at_the_fork and namecoin_latest_commit.
But, so many things have changed that I've put that idea aside, tonight.

I'm now starting a more step by step approch : take current namecoin & integrate all bitcoin commits one by one since version 0.3.24rc2.
This will be slow, repetitive, instructive on how bitcoin works now but I hope, in the end, more easy.
This represents 3400 commits to merge one by one, fix conflicts & check if it compiles (most slower part...)

Current work is available here and will be updated "regularely" :
https://github.com/khalahan/namecoin/tr ... merge_v0.8

Current work done represents 81 commits (around 2 hours of work, without any difficult commit).

Here is how I work :

One terminal proccess commits not existing in branch namecoin_merge_v0.8 and existing in branch bitcoin (I've imported a bitcoin branch in my namecoin respository).

Code: Select all

for CID in `tac <(git log namecoin_merge_v0.8..bitcoin --graph|grep -oE 'commit [a-f0-9]{40}'|cut -d' ' -f2)`; do git lg $CID -1; read -n1; git merge $CID; done
Each commit displays its infos, wait for a key press and then I can validate the commit.
If there are conflicts, I fix them in another terminal and commit.
Then, I can continue with merging the next commit.

On another side, I have gitk (gui tool to show commits/logs), that list each commits of the bitcoin branch with diffs, then I can see before merging the commit if it concerns namecoin or not, may provoke conflicts, edit a lot of .h files which will slow compilation, etc.

Now, some questions :
- Do you think it is possible to work at several on this (work is sequential) ? Someone interested ?
- How much frenquently should I try to compile namecoind ?
- How much frenquently should I try launch namecoind to check if things are ok ?
- Which tests in namecoind should I do ?

So ?

[edit] 2013-06-24 - Version 0.4.0 :
Here are 32 & 64 bits releases for linux (compiled on debian6) :
http://dot-bit.org/files/temp/namecoind64_0.4.0
http://dot-bit.org/files/temp/namecoind32_0.4.0

See : https://dot-bit.org/forum/viewtopic.php?p=4424#p4424
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Re: Rebasing namecoin on bitcoin 0.8

Post by khal »

I've worked a bit tonight (too hot to sleep in France...) and I'm now at 150 commits.
Last one added the class CBitcoinAddress that replace usage of string in application.

Namecoin makes an extended usage of strings for that, should I update Namecoin code to use CBitcoinAddress or not (similar cases may happen in the future) ? I would prefer to update Namecoin code too, code would be clearer, but, a lot of work :p

I'll release a binary when I'll reach bitcoin version 0.4, to allow people to test it :p
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

moa
Posts: 255
Joined: Mon May 23, 2011 6:13 am

Re: Rebasing namecoin on bitcoin 0.8

Post by moa »

khal: sounds like a lot of lonely work.

I suggest you at minimum make a release at same points when bitcoin released, i.e. 0.4, 0.5, 0.6, 0.6.1 ... etc.

When you get to these 'check points' maybe you can email/notify me somehow and I can pull that branch in and test build/run?

Edit: you are also going to have to consider what is going to happen when you get to the point in the bitcoin dev. when bitcoin-qt was merged and when you want to merge namecoin-qt of snailbrain's ...

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: Rebasing namecoin on bitcoin 0.8

Post by phelix »

Is it really worth it? It might be more effective to focus only on important / security relevant issues and in parallel on the next big thing: a kick ass namecoin based project.

Bitcoin devs will continue to produce millions of small commits that are barely relevant for us...


Also: https://bitcointalk.org/index.php?topic ... #msg987844

Not sure if this really is a good idea but there is a diff module for python... :mrgreen:
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Re: Rebasing namecoin on bitcoin 0.8

Post by khal »

phelix wrote:Is it really worth it? It might be more effective to focus only on important / security relevant issues and in parallel on the next big thing: a kick ass namecoin based project.

Bitcoin devs will continue to produce millions of small commits that are barely relevant for us...
There are several reasons to do that :
- really better performances
- removes most deadlock + new feature to track them (DEBUG_LOCKORDER)
- new index (Unspent TX Ouput list) using levelDB : can be used to replace the current index of names (based on BerkeleyDB : not suited for data removal => all names are stored, even old one/expired, to enable name_history. But this will not be usable at middle term.)
- unit tests
- benefit from all new things existing in bitcoin (multisig, p2sh, light nodes with bloom filtering, future pruning and so much other things that can be useful)
- and a lot of things I still haven't discovered...


An example of vulnerability : BIP-0050
https://en.bitcoin.it/wiki/Common_Vulne ... s#BIP-0050
Fixed in version 0.4.9rc2 <= Are you able to find this version/tag/commit ? Not me.
I guess only updated binary are delivered, so we will need an access to the unrevealed informations/commits from bitcoin developpers (should be possible, all will be required whatever version we are based on).


Also, I plan to do this as a background task. It is not the priority, but it is important to do imo, for the midlle/long term developpement of namecoin.
Maybe this could be skiped if we would have several and regular namecoin c++ coders ? (with a code based on bitcoin 0.8, we could get some coders interested by bitcoin first, into namecoin)

phelix wrote:Also: https://bitcointalk.org/index.php?topic ... #msg987844

Not sure if this really is a good idea but there is a diff module for python... :mrgreen:
Namecoin has a lot more differences with bitcoin compared to other alt-coins :p
Try this to see all differences from the version pre-namecoin to latest namecoin release (10k lines + hook inside the bitcoin code. Thats why I changed the way to merge with bitcoin 0.8 :p) :

Code: Select all

git diff --color 9390431ce49893cbdf23846edb4bdf72b3d4e830..82c140e7ceb6d49b2ec07613185c1fd3d0b832d6
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Re: Rebasing namecoin on bitcoin 0.8

Post by khal »

I've reached bitcoin commit for version v0.4.00rc2 ! :)

It has been tested by randy-waterhouse :
randy-waterhouse wrote:I built/run a5631050c48b29bfb8c168baf5a97e36d22f9487 on the merge_v0.8 branch ... seems okay with minimal testing, launches, connects, downloads blocks, getinfo, name_scan, stop all work ...
I'll reach version v0.4.0 soon :p (changelog for bitcoin v4.0 : https://en.bitcoin.it/wiki/Changelog#0.4.0.5B14.5D)

I'll release binary for linux 32/64 of v0.4.0.

Some changes occured in getwork/getworkaux and I would like to verify everything is still working.
I'll need some help from experimented merged mining miner.
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

snailbrain
Posts: 309
Joined: Tue Jul 19, 2011 9:33 pm

Re: Rebasing namecoin on bitcoin 0.8

Post by snailbrain »

khal wrote:I've reached bitcoin commit for version v0.4.00rc2 ! :)

It has been tested by randy-waterhouse :
randy-waterhouse wrote:I built/run a5631050c48b29bfb8c168baf5a97e36d22f9487 on the merge_v0.8 branch ... seems okay with minimal testing, launches, connects, downloads blocks, getinfo, name_scan, stop all work ...
I'll reach version v0.4.0 soon :p (changelog for bitcoin v4.0 : https://en.bitcoin.it/wiki/Changelog#0.4.0.5B14.5D)

I'll release binary for linux 32/64 of v0.4.0.

Some changes occured in getwork/getworkaux and I would like to verify everything is still working.
I'll need some help from experimented merged mining miner.
amazing .. very good of you to do this Khal

moa
Posts: 255
Joined: Mon May 23, 2011 6:13 am

Re: Rebasing namecoin on bitcoin 0.8

Post by moa »

khal : built/run commit fa6526240e74b18e0d760f628643ca6ea846d76e

minimal testing;
launches, connects, ACCEPT block, help, getinfo, listtransactions, name_show, name_list, name_scan, listaccounts, getaddressbyaccount, stop all appear to function without error.

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Re: Rebasing namecoin on bitcoin 0.8

Post by khal »

moa, thanks for testing :)

You have tested the v0.4.0, which has required to merge 468 commits from bitcoin :p
That work started from commit 82c140e7ceb6d49b2ec07613185c1fd3d0b832d6.


Here are 32 & 64 bits releases for linux (compiled on debian6) :
http://dot-bit.org/files/temp/namecoind64_0.4.0
http://dot-bit.org/files/temp/namecoind32_0.4.0


This release supports :
- The "new" wallet encryption feature
- The DEBUG_LOCKORDER flag (need to be added in the makefile it seems)
- Fix several bitcoin-becomes-unresponsive bugs due to multithreading deadlocks.
- Optimize database writes for large (lots of inputs) transactions (fixes a potential denial-of-service attack)

I can provide some testnet coins for people that want to help testing this release :p


Things that need to be tested :
- name_*
- everything with an encrypted wallet
- getauxwork for merged mining

Notes of the bitcoin v0.4.0 release : https://en.bitcoin.it/wiki/Changelog#0.4.0.5B14.5D

This release has a bug which may lead to some unencrypted privates keys (see https://bitcointalk.org/index.php?topic=51604.0). So, wait for version 0.4.1 before considering the wallet encryption feature as secured :p (there is no tag for v0.4.1 and further until v0.5.0 and don't know why but next commits are about bitcoin-qt, should be for version 0.4.6...)

From now, I'll stop merging commits and focus on other namecoin tasks (like nmcontrol :p). This is a very long task, and I think it should not take all my time at once :p.

I'll do tests on my side of course, and we will see what we can do with this work too (integrate into namecoin-qt ? make it a new official branch to replace what is currently namecoin/master ?).
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

khal
Site Admin
Posts: 708
Joined: Mon May 09, 2011 5:09 pm
os: linux

Re: Rebasing namecoin on bitcoin 0.8

Post by khal »

There is a bug with name_new : namecoind crash with each name_new, with encrypted and unencrypted wallet.

There is no problem in name_new until this :
string strError = pwalletMain->SendMoney(scriptPubKey, MIN_AMOUNT, wtx, false);

Here is the backtrace :
#0 DecodeBase58 (psz=0x0, vchRet=...) at base58.h:78
#1 0x00000000004278bd in DecodeBase58Check (psz=<optimized out>, vchRet=...) at base58.h:137
#2 0x00000000004a4fd0 in CBase58Data::SetString (this=0x7ffff4cc9d20, psz=<optimized out>) at base58.h:199
#3 0x00000000004d74e9 in CBitcoinAddress (pszAddress=0x0, this=0x7ffff4cc9d20) at base58.h:297
#4 CScript::GetBitcoinAddress (this=0xa414410) at script.h:639
#5 0x00000000004d1bb8 in CWallet::CreateTransaction (this=this@entry=0xa7b8170, vecSend=..., wtxNew=..., reservekey=..., nFeeRet=@0x7ffff4cca018: 500000) at wallet.cpp:985
#6 0x00000000004d23e8 in CWallet::CreateTransaction (this=this@entry=0xa7b8170, scriptPubKey=..., nValue=nValue@entry=1000000, wtxNew=..., reservekey=..., nFeeRet=@0x7ffff4cca018: 500000) at wallet.cpp:1038
#7 0x00000000004d25c2 in CWallet::SendMoney (this=0xa7b8170, scriptPubKey=..., nValue=1000000, wtxNew=..., fAskFee=false) at wallet.cpp:1106
#8 0x00000000004ed126 in name_new (params=..., fHelp=<optimized out>) at namecoin.cpp:1243
#9 0x0000000000495107 in ThreadRPCServer2 (parg=parg@entry=0x0) at rpc.cpp:2720
#10 0x0000000000496752 in ThreadRPCServer (parg=0x0) at rpc.cpp:2548
#11 0x00007ffff75aab50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007ffff6b55a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()
If someone want to investigate a bit I would appreciate the help :)


I've not tested name_firstupdate and name_update.

I've tested sendtoaddress, getnewaddress pr others with unencrypted & crypted wallet : ok
NamecoinID: id/khal
GPG : 9CC5B92E965D69A9
NMC: N1KHAL5C1CRzy58NdJwp1tbLze3XrkFxx9
BTC: 1KHAL8bUjnkMRMg9yd2dNrYnJgZGH8Nj6T

Register Namecoin domains with BTC
My bitcoin Identity - Send messages to bitcoin users
Charity Ad - Make a good deed without paying a cent

Post Reply