Page 1 of 1
Python Reproducible Builds
Posted: Mon Sep 28, 2015 9:31 pm
by phelix
Continued from
Security Implications of (Non)Reproducible Middleware Builds.
How to build the Python interpreter (and pyinstaller/alternative executable of e.g. NMControl).
I would support an NMDF bounty for this.
This might even be a good reason to switch to py3.
Re: Python Reproducible Builds
Posted: Mon Sep 28, 2015 10:26 pm
by josephbisch
The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can
see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
So the main reason that progress has stalled for Armory is that it would need to be ported to Python 3 first, because I was only able to get Python 3 to cross-compile, not Python 2. So it may be a while before any more progress is made with either the Windows build or OS X build of Armory, because they both require Python 3.
The general strategy for Armory is to use pyqtdeploy to create a static exe with Python, the various Python libraries that are required, and the Armory code itself. I went with that because it supported cross-compiling. In the case of pyinstaller, we could use it for the Windows build using wine, but we would have to do something else for OS X, because it doesn't support cross-compiling for OS X from Linux. After making sure that step works, we would need to take all the commands we would run to make the exe and put them in what is called a Gitian descriptor (Bitcoin Core can be used for a sample).
I can expand more on that later.
Re: Python Reproducible Builds
Posted: Tue Sep 29, 2015 7:44 am
by phelix
josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can
see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):
Code: Select all
I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
So the main reason that progress has stalled for Armory is that it would need to be ported to Python 3 first, because I was only able to get Python 3 to cross-compile, not Python 2. So it may be a while before any more progress is made with either the Windows build or OS X build of Armory, because they both require Python 3.
For NMControl this should not be too hard (particularly so if we remove all the NS stuff like Jeremy did
here).
The general strategy for Armory is to use pyqtdeploy to create a static exe with Python, the various Python libraries that are required, and the Armory code itself. I went with that because it supported cross-compiling. In the case of pyinstaller, we could use it for the Windows build using wine, but we would have to do something else for OS X, because it doesn't support cross-compiling for OS X from Linux. After making sure that step works, we would need to take all the commands we would run to make the exe and put them in what is called a Gitian descriptor (Bitcoin Core can be used for a sample).
I can expand more on that later.
Thanks for explaining. OS X building always seems to be even more difficult than Windows...
Re: Python Reproducible Builds
Posted: Sun Oct 18, 2015 6:16 am
by biolizard89
phelix wrote:josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can
see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):
Code: Select all
I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
Does that batch file run under Linux using only reproducibly buildable dependencies?
Re: Python Reproducible Builds
Posted: Sun Oct 18, 2015 5:17 pm
by josephbisch
biolizard89 wrote:phelix wrote:josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can
see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):
Code: Select all
I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
Does that batch file run under Linux using only reproducibly buildable dependencies?
The top of the batch file in question has the following:
echo.Build CPython from the command line. Requires the appropriate
echo.version(s) of Microsoft Visual Studio to be installed (see readme.txt).
echo.Also requires Subversion (svn.exe) to be on PATH if the '-e' flag is
echo.given.
So it does appear to be very Windows-centric (unsurprisingly as it is a Windows batch file, so normally you don't intend for it to be run on Linux).
We basically need msbuild to build the Visual Studio project.
Mono has an implementation of msbuild called
xbuild. Microsoft also has their official msbuild as open source it seems. There is an
xplat branch, which apparently runs on Linux. I don't know how you are expected to build it without access to a Windows machine, through, because it is itself a Visual Studio project.
I don't know how xbuild compares to msbuild.
Hopefully we can put together a shell script or something to build Python using xbuild or msbuild. I don't know if it will work though, much less be reproducible.
Re: Python Reproducible Builds
Posted: Mon Oct 19, 2015 8:09 am
by phelix
For me the moral of the story is that it should be possible some way. I will try to create an issue with Python and see if I can convince them to build reproducibly themselves. This is only a matter of time I think (but it may take very long).