comparison of minimal packaging between distribute and distutils2

vaab — Fri, 22 Feb 2013 15:27:14 +0000

In a pythonic mind, we shoud keep things simple as possible.

But when having to deal with distribution of your code, especially if it's ONE small python file, you'll might cringe when facing the boilerplate that is required. Even with distribute in place of setuptools.

And speaking of packaging system, python has a very complex history of them (between distutils, setuptools, distribute, distutils2), adding a considerable burden to get a clear idea of what system you should use, and get a complete documentation of one system. Especially when whole part of each system share code, ideas, compatibility but aims to be better in some obscure part for a newcomer.

Notice how this stackoverflow question about python packaging systems gets answered brilliantly and how concrete differences between all these packaging solutions weren't mentioned.

However, I wanted a dead simple way to distribute my single file package. And I found 2 ways:

one with distribute (the actually recommended packaging system),

one with distutils2 (the upcoming-but-not-finished replacement of distribute).

I realized that I could illustrate some simple differences for newcomers when facing the need to package a simple python file.

The goal

You have a file foo.py holding whatever python code you want. foo.py is alone in a directory.

Install-ability

You need to be able to type python setup.py install (or something similar) in the source directory and all the magic required will fire out to set things how they are supposed to be in your python libraries, allowing you to invoke import foo in any of your python file without any unfriendly exception being cast.

distributability

Next, you should be able to type python setup.py sdist register upload (or something similar) to build a source distribution, register the package to the PyPI, and send it. So that, finally, from any computer having access to internet, you could cast a pip install foo (or something similar) to get your package downloaded from the PyPI, and installed.

Distribute way

This is the simplest way I found to distribute a ONE small python file using distribute, which seems to be the current best solution waiting for distutils2.

Requirements

Make sure you have the last distribute installed. You can install it with this command:

curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py

setup.py file

Then create a setup.py file in the same directory than foo.py, containing:

from setuptools import setup
setup(
    ## Required for ``python setup.py install``
    name="foo_utils",
    version="0.1",
    py_modules=["foo"]

    ## Required for ``python setup.py sdist upload install``
)

The name will be your package name and version will be its version, these are used for naming the .egg and PyPI references.

The following is a less documented option and makes the solution I have found so small: py_modules is the python module you want to distribute. Typing foo targets the foo.py file.

Actually, complete reference of this trick can be found in distutils documentation, I couldn't find any reference to py_modules anywhere in the distribute documentation.

Tada !

The small setup.py you've written should allow you to type:

python setup.py install

To send it to PyPI, you'll need some other metadata as the author, summary, licence and maybe a small description. These are to be added in your setup.py file, and then:

python setup.py sdist register upload

... will send it to PyPI.

At last you'll then be able to install it from anywhere with:

pip install foo_utils

Distutils2 way

The major modification that you'll notice here is the fact that the configuration is not executable code anymore: setup.py is replaced by setup.cfg.

Requirements

You need to install distutils2:

pip install distutils2

If you have any trouble installing distutils2, you might be interested in the ending section about the truth about distutils2. You could also try something like that (adapt the version of course !):

cd /tmp
wget https://pypi.python.org/packages/source/D/Distutils2/Distutils2-1.0a4.tar.gz
tar xvzf Distutils2-1.0a4.tar.gz
cd Distutils2-1.0a4
python setup.py install

setup.cfg

In the same directory than foo.py, you can create setup.cfg containing:

[metadata]
name = foo_utils
version = 0.1

[files]
modules = foo

Notice that py_modules is in a files section and has been renamed to modules.

This first version of setup.cfg will allow you to achieve first goal of "install-ability". But you'll need more to get your code sent to PyPI:

[metadata]
name = foo_utils
version = 0.1.0
summary = A Simple example Foo program
description-file = README.rst CHANGELOG.rst

## sdist info
author = John Doe
author_email = john.doe@example.com
classifier =
    Development Status :: 3 - Alpha
    License :: OSI Approved :: GNU General Public License (GPL)

[files]
modules = foo_utils
extra_files =
    README.rst
    CHANGELOG.rst

Well, it's not that small, but I'm not sure what is actually really required, and I think a small summary or description won't hurt. Notice that we have to repeat our selves for the inclusion of the README.rst and CHANGELOG.rst files that are used in the description.

Of course, all details given hear are subject to changes in the upcoming releases of distutils2.

Tada !

The small setup.cfg you've written should allow you to type:

pysetup install

To reach the "install-ability" goal. And:

pysetup run sdist register upload

To send it to PyPI. Then you should be able to install it from anywhere with:

pysetup install foo_utils

Note

packages made with distutils2 won't be compatible with pip install PACKAGE commands as they do not provide any setup.py. You'll receive IOError: [Errno 2] No such file or directory.

If any of these commands don't work, jump ahead to the next section !.

The truth about distutils2

Before getting something actually working, I ran in multiple issues, and these are some of the trick I did to make it work.

First, get it installed:

cd /tmp
wget https://pypi.python.org/packages/source/D/Distutils2/Distutils2-1.0a4.tar.gz
tar xvzf Distutils2-1.0a4.tar.gz
cd Distutils2-1.0a4
python setup.py install

Then, I proceeded towards the distribute installation:

$ pysetup foo_utils
No handlers could be found for logger "distutils2"

Hum, after hacking into the distutils2 code, I've found that distutils hadn't any default logger, once one set, it was only to realize that it wanted to display a simple error to tell me that I messed with my arguments. The correct command is:

root@myhost$ pysetup install foo_utils
Checking the installation location...
Unable to write in "/usr/lib/python2.7/site-packages". Do you have the permissions ?
root@myhost$

What are you talking about ?! I'm root ! After looking around, I found there were no such directory site-packages in my testing VM, which should mimick an Ubuntu 12.04 standard install. I pursued by:

root@myhost$ mkdir /usr/lib/python2.7/site-packages
root@myhost$ pysetup install foo_utils
Checking the installation location...
Getting information about 'foo_utils'...
u'python-debian': u'0.1.21ubuntu1' is not a valid version (field 'Version')
u'python-apt': u'0.8.3ubuntu7' is not a valid version (field 'Version')
u'ufw': u'0.31.1-1' is not a valid version (field 'Version')
u'Landscape Client': u'12.05' is not a valid version (field 'Version')
Installing u'foo_utils' 0.1.0...
running install_dist
running build
running build_py
running install_lib
byte-compiling /usr/lib/python2.7/site-packages/foo.py to foo.pyc
running install_distinfo
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/METADATA
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/INSTALLER
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/REQUESTED
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/RECORD
root@myhost$

Success ! I then did the test to see if I could "import" my lib:

root@myhost$ python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named foo
>>>

Failure ! A quick check to sys.path will confirm that if the "site-package" was not existent, it wasn't included in the sys.path... which makes sense. I'm not aware of all these details, but why distutils2 then tries to install packages in this directory ?

Anyway, if I want to make sure that site-packages gets in sys.path, I can follow superuser advice:

echo ../site-packages > /usr/lib/python2.7/dist-packages/site-packages.pth

This will easily add site-packages directory system wide.

Conclusion

Is the packaging system of python dead-simple for one file ? Well, yes, once a straightforward recipe was discovered painfully.

Is it easy to find clear and unambiguous documentation ? Not really.

Is distutils2 finished ? Nope. Did you try with last available code from hg.python.org ? Yes.

Do you still need to use distribute ? Yes.

Any comments ?

Valentin's Lab » distutils2

comparison of minimal packaging between distribute and distutils2

The goal

Install-ability

distributability

Distribute way

Requirements

setup.py file

Tada !

Distutils2 way

Requirements

setup.cfg

Tada !

The truth about distutils2

Conclusion