Start a private pypi server

Start a pypi server with a docker container (yen3/pypiserver)

mkdir -p /tmp/pypi/packages
docker run -d -p 8080:8080 -v /tmp/pypi/packages:/data/packages yen3/pypiserver

Setting for the pypiserver

  • Add $HOME/.pypirc

    [distutils]
    index-servers = private_pypi_server
    
    [private_pypi_server]
    repository: http://localhost:8080
    username: admin
    password: admin
    
  • Add $HOME/.pip/pip.conf (Maybe you have to run mkdir $HOME/.pip first)

    [global]
    extra-index-url = http://localhost:8080
    trusted-host = localhost
    
  • Add $HOME/.pydistutils.cfg

    [easy_install]
    index-url = http://localhost:8080
    

Install package from server

  • With setting - as normal

    pip install <package-name>
    easy_install <package-name>   # If the package has egg only
    
  • Without any setting - If you don't add $HOME/.pypirc and $HOME/.pip/pip.conf. It's useful when you want to install some packages in docker container and the container has no setting.

    pip install --extra-index-url http://localhost:8000 --trusted-host localhost <package-name>
    easy_install --index-url=http://localhost:8000 <pakcage-name>
    

Package and Upload python programs/libraries

Package your project

Assume you already have a python library/program and ready to package.

In the following context, the security concern is to package it without source code.

There are several ways to build python package. By security concern, the package commands can be divided into

  • python setup.py sdist: Source package. Including everyting
  • python setup.py bdist_wheel: Binary package. Including all *.py files and compiled c extensions (*.so, *.o ... etc)
  • python setup.py bdist_egg --exclude-source-files: Binary package. Including all *.pyc files and compiled c extensions

How do you choose for your project?

  • If you have no security concern, use python setup.py sdist (or python setup.py bdist_wheel)
  • If you have the security concern, use python setup.py bdist_egg --exclude-source-files. But the output package is depndent on python version.
    • If you choose the way, the package can only be installed through easy_install
    • The other possible way is to use python setup.py bdist_wheel. After all packages are installed, run python3 -m compileall -b . in source folder and remove all *.py files.

Python2 or Python3 pakcage ?

In the previous section, the python version will decide the output package version. If your python is python2/python3, it would generate python2/python3 package (except sdist package). The package program would package your project without any checks (e.g. syntax check). You have to check everything is ok.

If you run python3 to package a python2 package, it would not cause any fail. But your user would pain for the package.

Universal package

It's possible to build a package both supported python 2.7 and 3 but the assumption is that you have to take care everything about source code compatibility in py2.7 and py3. If you make sure the source code in the package can both run in these versions smoothly. You can package with the command

python setup.py bdist_wheel --universal

Or add these contents in setup.cfg

[bdist_wheel]
universal=1

You can get more details from offical documents

Upload your package

  1. Assume there is a pypi server and it's named private_pypi_server in $HOME/pip/pip.conf
  2. The commands to upload your package are the same as package command but add upload -r private_pypi_server. e.g.
    • python setup.py sdist upload -r private_pypi_server
    • python setup.py bdist_wheel upload -r private_pypi_server
    • python setup.py bdist_egg --exclude-source-files upload -r private_pypi_server
Remove your uploaded package
  • Based on the current pypiserver's structure, all files are on /share/private_pypi_server/public/yen3/pypi. If you uploaded wrong packages or any other needs, you can find the file in the folder and remove it.

Q&A

  1. I can not execute python setup.py bdist_wheel
    • Install wheel (pip install wheel) and run again.
    • If the package define the run command, you have to check setup.py to get more detials.
  2. What's difference between egg and wheel?
    • See the offical docs: https://packaging.python.org/discussions/wheel-vs-egg/
  3. I upload and overwrite the package with the same version but pip installs the previous package.
    • When you package your project, remember to remove these files/folders
      • *.pyc
      • build/
      • dist/
      • __pycache__/
    • Remove the package first pip uninstall <package-name> and
      • Remove $HOME/.cache/pip
      • Try pip install -U --no-cache-dir <package-name>
    • If the problem is still unsolved
      • Roll your package version
      • Remove your python and reinstall (If you use pyenv, it's easy to resintall a new python)

Reference

  • Offical document
  • https://blog.zengrong.net/post/2169.html
  • https://stackoverflow.com/questions/5730686/installing-only-pyc-python-compiled-with-setuptools
  • https://blog.ionelmc.ro/presentations/packaging/#slide:1