Valentin's Lab » comp

Fixing Windows Python 2.7 unicode issue with subprocess’s Popen.

vaab — Thu, 16 Mar 2017 10:22:49 +0000

TL;DR

Fixing Windows Python 2.7 unicode issue with subprocess.Popen being unable to properly send command lines to the system.

Here is the python code fixing unicode issue

Details

What

Python 2.7 is plagued by bugs that won't be fixed.

One of them is that you can't use unicode chars on sub process call command line via subprocess.Popen(..) in windows platform.

This code fixes Popen(..) in python 2.7 under windows. This allows your code simply work on python 2.7 with little changes.

How

The following code leverage ctypes to call the CreateProcessW(..) function of the windows C API. This function should have been used in cPython 2.7 but wasn't. This is the core reason why Python 2.7 on windows does not support unicode command line.

This is how Python 3.0+ works.

The code

This code was not thoroughly tested and I'm posting it on a gist so I can update it and people can comment if they run into any issues.

So here is the python code fixing unicode issue.

A test ?

As a test, we'll create a small test.py python file that you can call with the new subprocess.Popen(..) provided in the Gist.

To be complete, we'll need another fix of python 2.7 unicode support: recipe to read the arguments in unicode.

Here's the full code for the testing:

with subprocess_fix module being the code in the Gist. Correctly sending the full unicode command line to the system. This is thus used on the calling side.
And commandline_fix module being the recipe to decode the current programs unicode command line with a final added sys.argv = win32_unicode_argv() statement. This is thus used on the called side.

The caller side (named test.py):

# -*- coding: utf-8 -*-

from subprocess import PIPE
from subprocess_fix import Popen

def indent(text, chars=" ", first=None):
    if first:
        first_line = text.split("\n")[0]
        rest = '\n'.join(text.split("\n")[1:])
        return '\n'.join([(first + first_line).rstrip(),
                          indent(rest, chars=chars)])
    return '\n'.join([(chars + line).rstrip()
                      for line in text.split('\n')])

p = Popen(u"python reading.py ć", shell=True,
stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate()

if p.returncode != 0:
    print("errlvl: %s" % p.returncode)

if err:
    print("stderr:\n%s" % indent(err, " | "))
if out:
    print("stdout:\n%s" % indent(out, " | "))

The called side (named reading.py):

# -*- coding: utf-8 -*-

import sys

import commandline_fix

print "command line received (repr): %r" % (sys.argv, )
print "command line received (str): %s" % (u" ".join(sys.argv), )

test.py uses the fixed Popen(..) to call reading.py in a subprocess and it specifies a unicode character in the command line arguments. reading.py will simply read the unicode command line and print it to the console.

To run the test:

python test.py

Notice that depending on your active charset (you can check with chcp), the console might fail to display the character correctly. You might need then to do a chcp 65001.

trapping SIGINT, SIGQUIT in asynchronous tasks

vaab — Sat, 30 Jul 2016 10:36:31 +0000

Scattered in bash documentation lies the truth about SIGINT and SIGQUIT signals being mysteriously ignored and non-trappable in background processes.

This has puzzled many, and will probably continue to do so.

Traps

Here's my try to unveil the terrible truth.

Let this script separate all cases and enlighten us:

cat <<'EOF' > /tmp/test.sh
#!/bin/bash
traps_show_and_test() {
    local label="$1"
    echo; echo "TRAPS from $label:"; trap
    trap "echo modified by $label" SIGINT  ## <-- MODIFICATION
    echo "TRAPS from $label (after modification attempt):"; trap
}
export -f traps_show_and_test

traps_show_and_test main
{ traps_show_and_test subshell; }
( traps_show_and_test subparen )
bash -c 'traps_show_and_test bash_subprocess'

traps_show_and_test job & wait
{ bash -c 'traps_show_and_test subshell_job'; } & wait
( bash -c 'traps_show_and_test subparen_job' ) & wait
bash -c 'traps_show_and_test bash_job' & wait

echo
echo FINAL main traps:
trap

EOF
chmod +x /tmp/test.sh
/tmp/test.sh

Running the previous script in bash (version is 4.3.46(1)-release) gives the following:

TRAPS from main:
TRAPS from main (after modification attempt):
trap -- 'echo modified by main' SIGINT

TRAPS from subshell:
trap -- 'echo modified by main' SIGINT
TRAPS from subshell (after modification attempt):
trap -- 'echo modified by subshell' SIGINT

TRAPS from subparen:
trap -- 'echo modified by subshell' SIGINT
TRAPS from subparen (after modification attempt):
trap -- 'echo modified by subparen' SIGINT

TRAPS from bash_subprocess:
TRAPS from bash_subprocess (after modification attempt):
trap -- 'echo modified by bash_subprocess' SIGINT

TRAPS from job:
trap -- 'echo modified by subshell' SIGINT
TRAPS from job (after modification attempt):
trap -- 'echo modified by job' SIGINT

TRAPS from subshell_job:
TRAPS from subshell_job (after modification attempt):
trap -- 'echo modified by subshell_job' SIGINT

TRAPS from subparen_job:
TRAPS from subparen_job (after modification attempt):
trap -- 'echo modified by subparen_job' SIGINT

TRAPS from bash_job:
trap -- '' SIGINT
trap -- '' SIGQUIT
TRAPS from bash_job (after modification attempt):
trap -- '' SIGINT
trap -- '' SIGQUIT

FINAL main traps:
trap -- 'echo modified by subshell' SIGINT

Let's highlight important points:

Foreground Jobs
- subshells from {..} and (..) inherit their traps from the main scope.
- only {..} can modify the main scope.
- subprocesses do not inherit its trap from its parent process
- they all can modify at least locally their traps
Background Jobs
- bash function inherits its traps from main scope but can't modify it (like (..))
- processes in {..} & and (..) & do not inherit traps
- direct background subprocess have default signed traps that are umodifiable

Bash documentation

As a reference, here are the documentation from bash related to these behaviors:

Process group id effect on background process (in Job Control section of doc):

[...] processes whose process group ID is equal to the current terminal process group ID [..] receive keyboard-generated signals such as SIGINT. These processes are said to be in the foreground. Background processes are those whose process group ID differs from the terminal's; such processes are immune to keyboard-generated signals.

Default handler for SIGINT and SIGQUIT (in Signals section of doc):

Non-builtin commands run by bash have signal handlers set to the values inherited by the shell from its parent. When job control is not in effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to these inherited handlers.

and about modification of traps (in trap builtin doc):

Signals ignored upon entry to the shell cannot be trapped or reset.

docker update, or how to incrementally build images without Dockerfile.

vaab — Wed, 28 Jan 2015 02:20:56 +0000

This handy little script covers an important missing feature in current docker's tool set.

I needed to build incrementally my images:

From a given image, I want full script control (un-tamed shell scripts) and access to hosts files to do whatever I want to do. At the end, I need to commit the filesystem (and only the filesystem).

docker-update manage all this:

docker-update my-docker-image -v /srv/files:/mnt/files <
A last thing about the docker-update script, is that it
tries to be clever (I know that's often the beginning of hell):
if you apply the same code to the same image ID, it won't execute
it, but will use the previous result instead.
You can disable the cache by inserting # docker: ALWAYS in
your bash code.
You can look at the source code of docker-update, or directly
download docker-update.

How does it work
It use docker run --entrypoint /bin/bash IMAGE on the bash
script, then docker commit.
Well that's the general
outline. In the detail, docker current shortcomings are making this idea much more
complex than it should.
The first docker run will modify ENTRYPOINT and CMD of image,
so when you commit, you break your previous values. So
docker-update needs to save them, and push another commit to
set them back.


docker-update shortcomings
As a quick and dirty solution, this script has some shortcomings
and sharp corners.  Some of these could easily repaired, and
additions are welcome.

NUL characters in your code
Any NUL character from your bash script will be removed prior to be run.
That's a direct consequence of running bash -c "$code". There are probably
better ways to do this.


Complex CMD and ENTRYPOINT
I rely on the fact that the output format of {{json .Config.Cmd}} templating system
is directly evaluable in the Dockerfile as arguments to CMD and ENTRYPOINT.
I'm pretty sure that this is bad. But it works in simple cases.


Caching
Remember that by default, your code will be checked wether it was already
run on the same image (identified by it's ID). If you don't want that to
happen, include # docker: ALWAYS.
For instance:
docker-update MYIMAGE <


docker-update options
They are sent directly to docker run command, and you can easily break
the command. It's primarily meant to be used with the -v option.

bash lore: how to properly parse NUL separated fields

vaab — Sat, 03 Jan 2015 09:26:28 +0000

As a lot of other part of bash, this is black magic.

Lets suppose a friendly command that spits out NUL separated fields (as find -print0, shyaml get-values-0, ...). Which - may I insist - is the recommended way to communicate wild binary data in a solid way in bash.

How would you parse adequately each individual records by paquets ?

For the purpose of demonstration, lets use the fixed content of a simple data.bin file as our NUL-separated input:

cat < /tmp/data.bin
a:1:b:2 3:c:4
  5:d:6\n7:e::f:9
EOF

Let's verify that we have our NUL bytes:

$ cat /tmp/data.bin | hexdump -v -e '/1 "%02X "'
61 00 31 00 62 00 32 20 33 00 63 00 34 0A 20 20 35 00 64 00 36 5C 6E 37 00 65 00 00 66 00 39 0A

You have noticed that we have some values containing:

spaces (hex: 20),

line breaks (hex: 0A),

a \ followed by a n.

a 0 sized value

a final value 9 ending with a 0a and no final 00.

If using NUL separated fields is recommended, it's to support this kind of data.

I want the implementation of a function read-0 that would allow this type of interaction:

$ cat /tmp/data.bin | while read-0 f1 f2; do
    echo "f1: '$f1', f2: '$f2'"
  done
f1: 'a', f2: '1'
f1: 'b', f2: '2 3'
f1: 'c', f2: '4
  5'
f1: 'd', f2: '6\n7'
f1: 'e', f2: ''
f1: 'f', f2: '9
'

First try

Let's be naive, and we'll use read f1 f2:

$ cat /tmp/data.bin | while read f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
f1: 'a1b2', f2: '3c4'
f1: '5d6n7ef9', f2: ''

You can notice that:

NUL char where ignored for field separation

fields where separated upon consecutive space or return, it uses value stored in IFS environment variable.

their are only 2 lines because the \n was used to separate each record. We should use -d to specify the line delimiter.

Note that the NUL chars are also extracted out of the data as variables don't support the NUL char.

The \ was eaten, because read builtin parse and give it special meaning. We should use -r to avoid that.

But how should we provide the NUL delimiter to the read builtin ? knowing that you can't put NUL chars on the command line ? Hopefully I stumbled onto this blog post: http://transnum.blogspot.sg/2008/11/bashs-read-built-in-supports-0-as.html

Conclusion is that -d '' should be understood magically by bash read builtin to delimit lines with NUL characters.

Better try

Let's apply our new acquired knowledge by trying IFS=$'\0' read -d '' -r f1 f2:

$ cat /tmp/data.bin | while IFS=$'\0' read -d '' -r f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
f1: 'a', f2: ''
f1: '1', f2: ''
f1: 'b', f2: ''
f1: '2 3', f2: ''
f1: 'c', f2: ''
f1: '4
  5', f2: ''
f1: 'd', f2: ''
f1: '6\n7', f2: ''
f1: 'e', f2: ''
f1: '', f2: ''
f1: 'f', f2: ''

That's much better. But notice that:

we didn't get anything in $f2, that's normal: by specifying NUL as line delimiter (with -d '') and having NUL as field delimiter (IFS) we will be doomed to have one field per record. We will need to manage the repacking in a while loop. This doesn't sound too difficult.

where's the final field 0A ? Hum, as there is no NUL final character in the data, read returned errlvl 1 on this last field but filled correctly the variable. A simple echo $f1 prints 9 (if you use this form: while IFS='' read -d '' -r f1 f2; do echo "f1: '$f1', f2: '$f2'"; done < /tmp/data.txt to access variables of the while).

Final Implementation ?

So knowing this, here is the final implemetation of read-0:

read-0() {
    local eof
    eof=
    while [ "$1" -a -z "$eof" ]; do
        IFS='' read -r -d '' "$1" || eof=true
        shift
    done
    test "$eof" != true -o -z "$1"
}

Final ? It surely properly works for our specification test. But what happens if EOF happens before we have fed all the variables ?:

$ echo -n "a" | while read-0 f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
$

Nothing is spit out, despite the fact that we have sent a character.

This is now a specification issue: Do we want read-0 to return errorlevel 0 when it hits EOF while filing the variables ? Okay, but we said 0-sized string was a possible value... read-0 in the current specification knows it hit EOF while filling variables as your first variable can be the 0-sized string. We could make a special case, but I want to be able to distinguish a last empty element from an element.

That read-0, in the actual specification, can't do it. But we can offer a slight change in the way you build your while loop to allow that parsing.

Correct Implementation

To fill partial records, will need another specification change as current implementation will fail whenever it encounters EOF. This is an incompatible specification issue. Aside from this, we need also to take care to actually set the value of the remaining fields to the empty string. This will require to use another version of read-0:

read-0() {
    local eof
    eof=
    while [ "$1" ]; do
        IFS='' read -r -d '' -- "$1" || eof=true
        shift
    done
    test "$eof" != true
}

So this would work with read-0:

$ echo -n a | tr :  '\000' | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: ''

$ echo -n a: | tr :  '\000' | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: ''

Basically, this construct allows a last round in the loop after detecting EOF... and achieve the starting spec:

$ cat /tmp/data.bin | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: '1'
f1: 'b', f2: '2 3'
f1: 'c', f2: '4
  5'
f1: 'd', f2: '6\n7'
f1: 'e', f2: ''
f1: 'f', f2: '9
'

Trivial ?

Happy hacking.

bash lore: NUL character and variables

vaab — Sat, 03 May 2014 10:29:27 +0000

Bash variables are often thought as being able to store any binary content.

Please bear in mind that it can't store NUL character, and only this one.

Here you are:

$ ascii_table() { echo -en "$(echo '\'0{0..3}{0..7}{0..7} | tr -d " ")"; }
$ ascii_table | hd
00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000020  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000030  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000040  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000050  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000060  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000070  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000080  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000090  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000000a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000000b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000000c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000000d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000000e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000000f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000100

But:

$ echo -n "$(ascii_table)" | hd
00000000  01 02 03 04 05 06 07 08  09 0a 0b 0c 0d 0e 0f 10  |................|
00000010  11 12 13 14 15 16 17 18  19 1a 1b 1c 1d 1e 1f 20  |............... |
00000020  21 22 23 24 25 26 27 28  29 2a 2b 2c 2d 2e 2f 30  |!"#$%&'()*+,-./0|
00000030  31 32 33 34 35 36 37 38  39 3a 3b 3c 3d 3e 3f 40  |123456789:;<=>?@|
00000040  41 42 43 44 45 46 47 48  49 4a 4b 4c 4d 4e 4f 50  |ABCDEFGHIJKLMNOP|
00000050  51 52 53 54 55 56 57 58  59 5a 5b 5c 5d 5e 5f 60  |QRSTUVWXYZ[\]^_`|
00000060  61 62 63 64 65 66 67 68  69 6a 6b 6c 6d 6e 6f 70  |abcdefghijklmnop|
00000070  71 72 73 74 75 76 77 78  79 7a 7b 7c 7d 7e 7f 80  |qrstuvwxyz{|}~..|
00000080  81 82 83 84 85 86 87 88  89 8a 8b 8c 8d 8e 8f 90  |................|
00000090  91 92 93 94 95 96 97 98  99 9a 9b 9c 9d 9e 9f a0  |................|
000000a0  a1 a2 a3 a4 a5 a6 a7 a8  a9 aa ab ac ad ae af b0  |................|
000000b0  b1 b2 b3 b4 b5 b6 b7 b8  b9 ba bb bc bd be bf c0  |................|
000000c0  c1 c2 c3 c4 c5 c6 c7 c8  c9 ca cb cc cd ce cf d0  |................|
000000d0  d1 d2 d3 d4 d5 d6 d7 d8  d9 da db dc dd de df e0  |................|
000000e0  e1 e2 e3 e4 e5 e6 e7 e8  e9 ea eb ec ed ee ef f0  |................|
000000f0  f1 f2 f3 f4 f5 f6 f7 f8  f9 fa fb fc fd fe ff     |...............|
000000ff

So bash variables (and $(...) idiom) are reasonably strong: if you know your content hasn't any NUL characters you can safely use them. All this is probably linked to the fact that command line arguments cannot hold NUL characters neither, a common string termination in C programs.

If you really need to store full binary content in a bash variable, you should think about encoding it (base64, xxd, or any format of yours).

Please understand that bash might also be able to do what you want without using variables but only pipes (stdin and stdout). They won't suffer from this limitation of course.

comparison of minimal packaging between distribute and distutils2

vaab — Fri, 22 Feb 2013 15:27:14 +0000

In a pythonic mind, we shoud keep things simple as possible.

But when having to deal with distribution of your code, especially if it's ONE small python file, you'll might cringe when facing the boilerplate that is required. Even with distribute in place of setuptools.

And speaking of packaging system, python has a very complex history of them (between distutils, setuptools, distribute, distutils2), adding a considerable burden to get a clear idea of what system you should use, and get a complete documentation of one system. Especially when whole part of each system share code, ideas, compatibility but aims to be better in some obscure part for a newcomer.

Notice how this stackoverflow question about python packaging systems gets answered brilliantly and how concrete differences between all these packaging solutions weren't mentioned.

However, I wanted a dead simple way to distribute my single file package. And I found 2 ways:

one with distribute (the actually recommended packaging system),

one with distutils2 (the upcoming-but-not-finished replacement of distribute).

I realized that I could illustrate some simple differences for newcomers when facing the need to package a simple python file.

The goal

You have a file foo.py holding whatever python code you want. foo.py is alone in a directory.

Install-ability

You need to be able to type python setup.py install (or something similar) in the source directory and all the magic required will fire out to set things how they are supposed to be in your python libraries, allowing you to invoke import foo in any of your python file without any unfriendly exception being cast.

distributability

Next, you should be able to type python setup.py sdist register upload (or something similar) to build a source distribution, register the package to the PyPI, and send it. So that, finally, from any computer having access to internet, you could cast a pip install foo (or something similar) to get your package downloaded from the PyPI, and installed.

Distribute way

This is the simplest way I found to distribute a ONE small python file using distribute, which seems to be the current best solution waiting for distutils2.

Requirements

Make sure you have the last distribute installed. You can install it with this command:

curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py

setup.py file

Then create a setup.py file in the same directory than foo.py, containing:

from setuptools import setup
setup(
    ## Required for ``python setup.py install``
    name="foo_utils",
    version="0.1",
    py_modules=["foo"]

    ## Required for ``python setup.py sdist upload install``
)

The name will be your package name and version will be its version, these are used for naming the .egg and PyPI references.

The following is a less documented option and makes the solution I have found so small: py_modules is the python module you want to distribute. Typing foo targets the foo.py file.

Actually, complete reference of this trick can be found in distutils documentation, I couldn't find any reference to py_modules anywhere in the distribute documentation.

Tada !

The small setup.py you've written should allow you to type:

python setup.py install

To send it to PyPI, you'll need some other metadata as the author, summary, licence and maybe a small description. These are to be added in your setup.py file, and then:

python setup.py sdist register upload

... will send it to PyPI.

At last you'll then be able to install it from anywhere with:

pip install foo_utils

Distutils2 way

The major modification that you'll notice here is the fact that the configuration is not executable code anymore: setup.py is replaced by setup.cfg.

Requirements

You need to install distutils2:

pip install distutils2

If you have any trouble installing distutils2, you might be interested in the ending section about the truth about distutils2. You could also try something like that (adapt the version of course !):

cd /tmp
wget https://pypi.python.org/packages/source/D/Distutils2/Distutils2-1.0a4.tar.gz
tar xvzf Distutils2-1.0a4.tar.gz
cd Distutils2-1.0a4
python setup.py install

setup.cfg

In the same directory than foo.py, you can create setup.cfg containing:

[metadata]
name = foo_utils
version = 0.1

[files]
modules = foo

Notice that py_modules is in a files section and has been renamed to modules.

This first version of setup.cfg will allow you to achieve first goal of "install-ability". But you'll need more to get your code sent to PyPI:

[metadata]
name = foo_utils
version = 0.1.0
summary = A Simple example Foo program
description-file = README.rst CHANGELOG.rst

## sdist info
author = John Doe
author_email = john.doe@example.com
classifier =
    Development Status :: 3 - Alpha
    License :: OSI Approved :: GNU General Public License (GPL)

[files]
modules = foo_utils
extra_files =
    README.rst
    CHANGELOG.rst

Well, it's not that small, but I'm not sure what is actually really required, and I think a small summary or description won't hurt. Notice that we have to repeat our selves for the inclusion of the README.rst and CHANGELOG.rst files that are used in the description.

Of course, all details given hear are subject to changes in the upcoming releases of distutils2.

Tada !

The small setup.cfg you've written should allow you to type:

pysetup install

To reach the "install-ability" goal. And:

pysetup run sdist register upload

To send it to PyPI. Then you should be able to install it from anywhere with:

pysetup install foo_utils

Note

packages made with distutils2 won't be compatible with pip install PACKAGE commands as they do not provide any setup.py. You'll receive IOError: [Errno 2] No such file or directory.

If any of these commands don't work, jump ahead to the next section !.

The truth about distutils2

Before getting something actually working, I ran in multiple issues, and these are some of the trick I did to make it work.

First, get it installed:

cd /tmp
wget https://pypi.python.org/packages/source/D/Distutils2/Distutils2-1.0a4.tar.gz
tar xvzf Distutils2-1.0a4.tar.gz
cd Distutils2-1.0a4
python setup.py install

Then, I proceeded towards the distribute installation:

$ pysetup foo_utils
No handlers could be found for logger "distutils2"

Hum, after hacking into the distutils2 code, I've found that distutils hadn't any default logger, once one set, it was only to realize that it wanted to display a simple error to tell me that I messed with my arguments. The correct command is:

root@myhost$ pysetup install foo_utils
Checking the installation location...
Unable to write in "/usr/lib/python2.7/site-packages". Do you have the permissions ?
root@myhost$

What are you talking about ?! I'm root ! After looking around, I found there were no such directory site-packages in my testing VM, which should mimick an Ubuntu 12.04 standard install. I pursued by:

root@myhost$ mkdir /usr/lib/python2.7/site-packages
root@myhost$ pysetup install foo_utils
Checking the installation location...
Getting information about 'foo_utils'...
u'python-debian': u'0.1.21ubuntu1' is not a valid version (field 'Version')
u'python-apt': u'0.8.3ubuntu7' is not a valid version (field 'Version')
u'ufw': u'0.31.1-1' is not a valid version (field 'Version')
u'Landscape Client': u'12.05' is not a valid version (field 'Version')
Installing u'foo_utils' 0.1.0...
running install_dist
running build
running build_py
running install_lib
byte-compiling /usr/lib/python2.7/site-packages/foo.py to foo.pyc
running install_distinfo
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/METADATA
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/INSTALLER
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/REQUESTED
creating /usr/lib/python2.7/site-packages/foo-utils-0.1.0.dist-info/RECORD
root@myhost$

Success ! I then did the test to see if I could "import" my lib:

root@myhost$ python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named foo
>>>

Failure ! A quick check to sys.path will confirm that if the "site-package" was not existent, it wasn't included in the sys.path... which makes sense. I'm not aware of all these details, but why distutils2 then tries to install packages in this directory ?

Anyway, if I want to make sure that site-packages gets in sys.path, I can follow superuser advice:

echo ../site-packages > /usr/lib/python2.7/dist-packages/site-packages.pth

This will easily add site-packages directory system wide.

Conclusion

Is the packaging system of python dead-simple for one file ? Well, yes, once a straightforward recipe was discovered painfully.

Is it easy to find clear and unambiguous documentation ? Not really.

Is distutils2 finished ? Nope. Did you try with last available code from hg.python.org ? Yes.

Do you still need to use distribute ? Yes.

Any comments ?

Emacs and flymake for python, javascript, php, rst

vaab — Thu, 20 Sep 2012 10:28:16 +0000

Emacs is a wonderfull editor, every sane people know this. Here's how to extend your flymake to some trendy languages.

You'll find here a general overview by answering question as What's flymake ? for emacs, and How does it work ?.

Following are the snippet of code to have nice flymake integration for:

Python

PHP

Javascript

ReSTructured Text

Happy coding !

What's flymake ?

flymake-mode in emacs is meant to "compile" your code while you are typing (no need to save) so as to give you interesting highlights on particular bits of code you've written.

If these can spot syntax errors, code analyzers can go much beyond and show you logical errors as unused variables, unreachable code, an so on. And there's more: you can be hinted when breaking code style conventions that you've chosen, or even code smell detectors (too much method in an object, too much local variable, ...)

How does it work ?

The big picture

flymake-mode is very simple, as you type it'll create a copy of your current emacs buffer in a file usualy called -flymake..

Then, it'll launch an executable on this file and will collect it's standard output.

To work out of the box, the format should be something like this:

myfile.myext:: : error message

This is a typical output of my pycheckers script which combines pep8 code style checker and pylint (syntax, logical, code style, code smell checking):

converter.py:38: ERROR: E501 (pep8) line too long (141 characters)
converter.py:130: ERROR: E303 (pep8) too many blank lines (2)
converter.py:156: WARNING: W0212 (pylint) Access to a protected member _fields_id of a client class
converter.py:185: WARNING: R0911 (pylint) Too many return statements (9/6)

emacs will then use this output to highlight WARNING and ERROR message in two different colors.

Note

Besides helping you in getting better code every day, having the same format, one by line, with clear tagging of the error type and the analyzer can be extremely usefull in continuous integration to get a valuable metric on code quality.

What do I need ?

You'll need to enable flymake once for all by adding the following line to your ~/.emacs, which will launch flymake whenever you'll open a file:

(add-hook 'find-file-hook 'flymake-find-file-hook)

Then to set up flymake-mode for a langage you'll need to:

1. add a specific snippet of code in ~/.emacs to configure the analyzer (ie: to set the actual name of the executable analyzer and its command line arguments if any).

2. create the executable code as it often require some tweaking to get the correct output. This stage is optional if your analyzer output is already in the correct format.

Ok, where's the code ?

Python

Analyzer

My pycheckers code analyzer is actually written in python and will call pep8 and pylint and ensure that the output format is correct. These two analyzers are available on PyPI and thus are installable thanks to:

pip install pylint
pip install pep8

Please check the analyzer code at: https://gist.github.com/3754270

I typically put the analyzer in ~/bin/pycheckers and ensure that it is executable.

Emacs

Here's the emacs snippet:

;; Python flymake configuration

(when (load "flymake" t)
  (defun flymake-pycheckers-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                    'flymake-create-temp-inplace))
        (local-file (file-relative-name
                     temp-file
                     (file-name-directory buffer-file-name))))
      (list "~/bin/pycheckers.py"  (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
            '("\\.py\\'" flymake-pycheckers-init)))

For the lazier

This snippet will do the complete job for you:

mkdir -p ~/bin &&
cd ~/bin &&
wget https://raw.github.com/gist/3754270/a4fecffaff6ab702f5c8d37332afc90f6461e40b/pycheckers.py &&
chmod +x pycheckers.py &&
cat <> ~/.emacs

;; Python flymake configuration

(when (load "flymake" t)
  (defun flymake-pycheckers-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                    'flymake-create-temp-inplace))
        (local-file (file-relative-name
                     temp-file
                     (file-name-directory buffer-file-name))))
      (list "~/bin/pycheckers.py"  (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
            '("\\.py\\'" flymake-pycheckers-init)))

EOF

You'll need to restart emacs for this to work. (Or highlight the added snippet in the end of your .emacs and call M-x eval-region)

Php

Analyzer

I've also a phpcheckers, even if it is very small, I feel better to know that I could grep out some specific warnings or add another sub-analyzer quite easily, and feel free to add your own include path as you wish in the 'packages-path' arguments:

#!/bin/bash

phplint --modules-path ~/var/lib/phplint/modules \
        --packages-path /usr/share/php \
        --no-print-notices "$@"

So i'm using only phplint here.

Emacs

Here's the emacs snippet:

;; PHP flymake configuration

(when (load "flymake" t)
  (defun flymake-phplint-init()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                     'flymake-create-temp-inplace))
         (local-file (file-relative-name
                      temp-file
                      (file-name-directory buffer-file-name))))
      (list "phpcheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
             '("\\.php[345]?$" flymake-phplint-init))
  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.inc$" flymake-phplint-init)))

For the lazier

This snippet will do the complete job for you:

mkdir -p ~/bin &&
cd ~/bin &&
cat < ~/bin/phpcheckers &&
#!/bin/bash

phplint --modules-path ~/var/lib/phplint/modules \
        --packages-path /usr/share/php \
        --no-print-notices "\$@"
EOF
chmod +x phpcheckers &&
cat <> ~/.emacs

;; PHP flymake configuration

(when (load "flymake" t)
  (defun flymake-phplint-init()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                    'flymake-create-temp-inplace))
        (local-file (file-relative-name
                     temp-file
                     (file-name-directory buffer-file-name))))
      (list "~/bin/phpcheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
            '("\\.php[345]?$" flymake-phplint-init))
  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.inc$" flymake-phplint-init)))

EOF

You'll need to restart emacs for this to work. (Or highlight the added snippet in the end of your .emacs and call M-x eval-region)

Javascript

Analyzer

I use a combination of jslint and jshint.

This is more work as jslint is widely criticized for behing too spotty on some conventions only shared by its author, I find it useful when tamed down to normal behavior.

Second, it's not so easy to get the correct format out of jslint. So I created a jslint-emacs executable that lies in ~/bin/jslint-emacs. Here's the code:

#!/bin/bash

filename="$1"

jslint "$filename" --nomen --maxerr 999999 |

## reformat from:
##
##  #45 Expected exactly one space between 'function' and '('.
##      init: function(parent, dataset, csv_import_record){ // Line 127, Pos 23
##
## to:
##
##  filename.js:94: WARNING: (jslint) Expected exactly one space between 'function' and '('.

         sed -r "N;s/^\s*#[0-9]+ (.*)\n\s+.*\/\/ Line ([0-9]+),.*/$filename:\2: WARNING: W000 (jslint) \1/" |

## Remove garbages lines that were not match by previous regex
## as their is a small header not matching regexp.

         grep -E "^$filename:[0-9]+:" |

         grep -v "at column.*not column.*." |
         grep -v "Expected '{' and instead saw 'return'" |
         grep -v "Expected exactly one space between ')' and 'return'" |
         grep -v "Combine this with the previous 'var' statement" |
         grep -v "Unexpected 'in'. Compare with undefined, or use .*"

## force errorlevel 0 so as to avoid emacs ``flymake-mode`` to complain
exit 0

I've found that jshint can use a specific reporter written in javascript, which seems perfect to get the correct ouput from it.

This is my ~/bin/jshint-emacs:

#!/bin/bash

jshint "$1" --reporter ~/.emacs.d/jslint-emacs-reporter.js

And of course, my ~/.emacs.d/jslint-emacs-reporter.js is available as a gist here: https://gist.github.com/3754379

It seems that this javascript reporter could be used also with jslint but I didn't figure out how to do so. (This was maybe because I've already written the sed/grep thing before).

Of course, this is my ~/bin/jscheckers:

#!/bin/bash

jslint-emacs "$1"
jshint-emacs "$1"

Emacs

Here's the emacs snippet:

;; Javascript flymake configuration

(when (load "flymake" t)
  (defun flymake-jslint-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                     'flymake-create-temp-inplace))
           (local-file (file-relative-name
                        temp-file
                        (file-name-directory buffer-file-name))))
      (list "jscheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.js\\'" flymake-jslint-init)))

For the lazier

This snippet will do the complete job for you:

mkdir -p ~/bin &&
cd ~/bin &&
cat < ~/bin/jslint-emacs &&
#!/bin/bash

filename="\$1"

jslint "\$filename" --nomen --maxerr 999999 |

## reformat from:
##
##  #45 Expected exactly one space between 'function' and '('.
##      init: function(parent, dataset, csv_import_record){ // Line 127, Pos 23
##
## to:
##
##  filename.js:94: WARNING: (jslint) Expected exactly one space between 'function' and '('.

         sed -r "N;s/^\s*#[0-9]+ (.*)\n\s+.*\/\/ Line ([0-9]+),.*/\$filename:\2: WARNING: W000 (jslint) \1/" |

## Remove garbages lines that were not match by previous regex
## as their is a small header not matching regexp.

         grep -E "^\$filename:[0-9]+:" |

         grep -v "at column.*not column.*." |
         grep -v "Expected '{' and instead saw 'return'" |
         grep -v "Expected exactly one space between ')' and 'return'" |
         grep -v "Combine this with the previous 'var' statement" |
      grep -v "Unexpected 'in'. Compare with undefined, or use .*"

## force errorlevel 0 so as to avoid emacs ``flymake-mode`` to complain
exit 0
EOF
chmod +x jslint-emacs &&
cat < ~/bin/jscheckers &&
#!/bin/bash

~/bin/jslint-emacs "\$1"
~/bin/jshint-emacs "\$1"

EOF
chmod +x jscheckers &&
mkdir -p ~/.emacs.d &&
cd ~/.emacs.d &&
wget https://raw.github.com/gist/3754379/3c43c50b4d85f50b8314af8e498b1b4f2cee58ea/jslint-emacs-reporter.js &&
cat <> ~/.emacs

;; Javascript flymake configuration

(when (load "flymake" t)
  (defun flymake-jslint-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                    'flymake-create-temp-inplace))
           (local-file (file-relative-name
                        temp-file
                        (file-name-directory buffer-file-name))))
      (list "jscheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.js\\'" flymake-jslint-init)))

EOF

You'll need to restart emacs for this to work. (Or highlight the added snippet in the end of your .emacs and call M-x eval-region)

ReSTructured Text

Analyzer

I was surprised to see that rst2html will make very decent rst analyzer.

rst2html is provided by python docutils package.

My ~/bin/rstcheckers:

#!/bin/bash

rst2html "$1" > /dev/null

Surprising isn't it ? As it isn't a code analyzer but the compiler itself, it does it's job really well, and in combination with flymake, it gives you a very convenient on-the-fly syntax checking.

Note that >/dev/null is required as rst2html jobs is to output HTML from your rst (which we don't want to see), and it casts errors on the standard error pipe.

Emacs

Here's the emacs snippet:

;; ReSTructured Text Flymake configuration

(when (load "flymake" t)
  (defun flymake-rst-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                     'flymake-create-temp-inplace))
           (local-file (file-relative-name
                        temp-file
                        (file-name-directory buffer-file-name))))
      (list "rstcheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.rst\\'" flymake-rst-init)))

For the lazier

This snippet will do the complete job for you:

mkdir -p ~/bin &&
cd ~/bin &&
cat < ~/bin/rstcheckers &&
#!/bin/bash

rst2html "\$1" > /dev/null
EOF
chmod +x rstcheckers &&
cat <> ~/.emacs

;; ReSTructured Text Flymake configuration

(when (load "flymake" t)
  (defun flymake-rst-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
                    'flymake-create-temp-inplace))
           (local-file (file-relative-name
                        temp-file
                        (file-name-directory buffer-file-name))))
      (list "rstcheckers" (list local-file))))

  (add-to-list 'flymake-allowed-file-name-masks
               '("\\.rst\\'" flymake-rst-init)))

EOF

You'll need to restart emacs for this to work. (Or highlight the added snippet in the end of your .emacs and call M-x eval-region)

Conclusion

I hope this will let you add some more language to your emacs flymake-mode. If you spot any errors, please let me know.

cleaned and released kal-shlib-* code and dd_rhelp

vaab — Mon, 30 Apr 2012 16:44:47 +0000

This is a long time since I wrote the first line of this shell script library which has followed me since then.

These library were managed on subversion, so this was the opportunity to move them toward GIT.

kal-shlib-core -- the core library loading mecanism
kal-shlib-common -- very common and basic functions
kal-shlib-pretty -- ANSI colored wrapper to get humble but pretty outputs
kal-shlib-shunit -- Shell Unit test framework

Same for dd_rhelp which received a small but constant flow of thanks, remarks, and patches from all over the world. And I wasn't always very reactive.

Hope this will help people mess around with the code as they wish.

SpamAssassin, Amavis and ubuntu

vaab — Mon, 02 Apr 2012 19:59:01 +0000

If running spamassassin (SA) with amavis without usage of spamd on ubuntu 10.04 and you feel that sa-learn isn't working right, here are a few tips.

I had mails that were false positive and I had to dig a little more than usual to get it working.

learning

Think that SA is executed by amavis user, so:

sudo -u amavis -H sa-learn --showdots {--ham|--spam} YOUR_MAIL_OR_MBOX_OR_MAILDIR

The trick is the sudo part, sa-learn docs can be found everywhere on the net.

checking

This is how to check how spamassassin will rate your mail:

cat badmail | sudo -u amavis -H spamassassin -t

Which is very usefull when used with the same mail, before and after learning.

statistics

Looking at statistics can be usefull to check that the number of spam and ham are consistent with your learning sessions:

sudo -u amavis -H sa-learn --dump magic | grep am

Note

Do not forget to restart amavis with /etc/init.d/amavis restart.

extracting a subpart of a git repository with history

vaab — Wed, 29 Jun 2011 08:10:24 +0000

I recently had to extract a subdirectory of an existing git repository towards its own new repository. And I wanted to keep all the commit information and history that was related to this directory.

EDIT: Even if the solution I used was simpler than I would have expected, I was far from knowing that GIT had a 'filter-branch' command which does the job perfectly well (and it is documented in ProGIT). Thanks Jon for the tip.

sketching the tools used

First step, is to retrieve only the commit Hashes related to this directory:

git log --format=%H -- MY_DIRECTORY

to filter out only the commit Hashes related to this directory. Then:

git format-patch HASH -1

will be used to output patch files with the commit information (subject, author...)

Then, with some sed you'll have to change the directories used in all the outputed patch:

sed -i s%MYOLDDIRECTORY%MYNEWDIRECTORY%g *.patch

The new directory in my case was "" (nothing) as it was in the root folder.

Finally, you'll have to apply your patches to your other repository with:

git am *.patch

Complete commands used

The complete commands that I've used:

cd PARENT_OF_MYDIRECTORY
git log --format=%H -- MYDIRECTORY | tac | \
    (i=0
     while read h; do
         i=$[$i+1]
         git format-patch $h -1 --start-number $i
     done)

This produces NNNN-*.patch files, one for each commit:

sed -i s%MYDIRECTORY%%g *.patch

This will update the paths in all patches:

mv *.patch MY_NEW_REPO
cd MY_NEW_REPO
git am *.patch

A quick check with gitk ensures me that message subject and commit were transfered ok.

Et voilà !