Python Reproducible Builds

Post Reply
phelix
Posts: 1625
Joined: Thu Aug 18, 2011 6:59 am

Python Reproducible Builds

Post by phelix » Mon Sep 28, 2015 9:31 pm

Continued from Security Implications of (Non)Reproducible Middleware Builds.

How to build the Python interpreter (and pyinstaller/alternative executable of e.g. NMControl).

I would support an NMDF bounty for this.

This might even be a good reason to switch to py3. :mrgreen:
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

josephbisch
Posts: 69
Joined: Sun Nov 23, 2014 3:34 pm
os: linux

Re: Python Reproducible Builds

Post by josephbisch » Mon Sep 28, 2015 10:26 pm

The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.

So the main reason that progress has stalled for Armory is that it would need to be ported to Python 3 first, because I was only able to get Python 3 to cross-compile, not Python 2. So it may be a while before any more progress is made with either the Windows build or OS X build of Armory, because they both require Python 3.

The general strategy for Armory is to use pyqtdeploy to create a static exe with Python, the various Python libraries that are required, and the Armory code itself. I went with that because it supported cross-compiling. In the case of pyinstaller, we could use it for the Windows build using wine, but we would have to do something else for OS X, because it doesn't support cross-compiling for OS X from Linux. After making sure that step works, we would need to take all the commands we would run to make the exe and put them in what is called a Gitian descriptor (Bitcoin Core can be used for a sample).

I can expand more on that later.

phelix
Posts: 1625
Joined: Thu Aug 18, 2011 6:59 am

Re: Python Reproducible Builds

Post by phelix » Tue Sep 29, 2015 7:44 am

josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):

Code: Select all

I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
So the main reason that progress has stalled for Armory is that it would need to be ported to Python 3 first, because I was only able to get Python 3 to cross-compile, not Python 2. So it may be a while before any more progress is made with either the Windows build or OS X build of Armory, because they both require Python 3.
For NMControl this should not be too hard (particularly so if we remove all the NS stuff like Jeremy did here).
The general strategy for Armory is to use pyqtdeploy to create a static exe with Python, the various Python libraries that are required, and the Armory code itself. I went with that because it supported cross-compiling. In the case of pyinstaller, we could use it for the Windows build using wine, but we would have to do something else for OS X, because it doesn't support cross-compiling for OS X from Linux. After making sure that step works, we would need to take all the commands we would run to make the exe and put them in what is called a Gitian descriptor (Bitcoin Core can be used for a sample).

I can expand more on that later.
Thanks for explaining. OS X building always seems to be even more difficult than Windows...
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

biolizard89
Posts: 1932
Joined: Tue Jun 05, 2012 6:25 am
os: linux

Re: Python Reproducible Builds

Post by biolizard89 » Sun Oct 18, 2015 6:16 am

phelix wrote:
josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):

Code: Select all

I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
Does that batch file run under Linux using only reproducibly buildable dependencies?
Jeremy Rand, Lead Namecoin Application Engineer
NameID: id/jeremy
DyName: Dynamic DNS update client for .bit domains.

Donations: BTC 1EcUWRa9H6ZuWPkF3BDj6k4k1vCgv41ab8 ; NMC NFqbaS7ReiQ9MBmsowwcDSmp4iDznjmEh5

josephbisch
Posts: 69
Joined: Sun Nov 23, 2014 3:34 pm
os: linux

Re: Python Reproducible Builds

Post by josephbisch » Sun Oct 18, 2015 5:17 pm

biolizard89 wrote:
phelix wrote:
josephbisch wrote:The main problem is that the process (for Windows at least) needs to be streamlined and fully automatable before we can even look at putting it in a Gitian descriptor to see if it builds reproducibly or if there is more work to be done. You can see the instructions in the initial comment and see how they are not fully automated. For a while I was having trouble getting the ssl module to work when cross compiling for Windows from Linux. But the last comments talk about how I got that working.
Hmm, according to Steve Dower there is a batch file for building Python on Windows... (I need to try this, would be great if it really worked):

Code: Select all

I have exactly one batch file to build the full span of releases for Windows, and I just run that. It's public and in the repo, so anyone else can also run it, they just won't get bit-for-bit identical builds because of timestamps, embedded paths, and certificates.
Does that batch file run under Linux using only reproducibly buildable dependencies?
The top of the batch file in question has the following:
echo.Build CPython from the command line. Requires the appropriate
echo.version(s) of Microsoft Visual Studio to be installed (see readme.txt).
echo.Also requires Subversion (svn.exe) to be on PATH if the '-e' flag is
echo.given.
So it does appear to be very Windows-centric (unsurprisingly as it is a Windows batch file, so normally you don't intend for it to be run on Linux).

We basically need msbuild to build the Visual Studio project.

Mono has an implementation of msbuild called xbuild. Microsoft also has their official msbuild as open source it seems. There is an xplat branch, which apparently runs on Linux. I don't know how you are expected to build it without access to a Windows machine, through, because it is itself a Visual Studio project.

I don't know how xbuild compares to msbuild.

Hopefully we can put together a shell script or something to build Python using xbuild or msbuild. I don't know if it will work though, much less be reproducible.

phelix
Posts: 1625
Joined: Thu Aug 18, 2011 6:59 am

Re: Python Reproducible Builds

Post by phelix » Mon Oct 19, 2015 8:09 am

For me the moral of the story is that it should be possible some way. I will try to create an issue with Python and see if I can convince them to build reproducibly themselves. This is only a matter of time I think (but it may take very long).
nx.bit - some namecoin stats
nf.bit - shortcut to this forum

Post Reply