This post is updated on 3rd of March 2016. If you would like to see what changed check github commit history
pip install can take quite a while when we try to install packages as big as scikit-learn or scipy from source. Especially if you’re installing it quite often within virtualenv and not on that powerful vagrant virtual machine.
wheel is “a built-package format for Python”.
This is the replacement for the egg format which never had proper pep specification. Wheel gives us an option to create binary “package” by ourself. If there already is
.whl uploaded to pypi
pip wheel will download and use it.
pip wheel can create binaries and with help of
devpi-builder it’s easy to create local mirror of compiled packages.
devpi-builder to build packages.
virtualenv for every
requirements.txt file for which I want to build wheels.
Install all dependencies from
I’m doing this step just as précaution because for example
numpy installed to build but
numpy is not added as dependency.
devpi is the easiest part:
To make it easier to start/stop service I also created user in the system called devpi and enabled
systemd to run processess for users even if they logged out.
.service file looks like this:
File was copied to
/home/$USER/.config/systemd/user/devpi.service and the service was started with
Now it’s time to configure our local mirror.
Building and uploading packages to devpi server
And now we have our locally available packages which we can use even if the internet (or pypi) is down.
From now you can go to devpi-setup-example repository and test it with vagrant or just see scripts with simple devpi installation.
Global pip configuration with fallback to pypi
And the last thing I wanted to do is to create pip config which can be used everywhere. When the local mirror is not available/dead/broken it’s going to download packages from pypi.
Where to put this
pip.conf you can find on pip documentation page
I’m using this method to build packages for python2 and python3. It takes about 3 hours right now for me so this is not the most efficient way.
This is why I’m going to write script taking all the dependencies across repositories, checks versions and builds all the things I need. It’s going to be totally for my use case but I think I’ll opensource it and maybe hack a little around pip APIs.