Valentin's Lab » tip

bash lore: how to properly parse NUL separated fields

vaab — Sat, 03 Jan 2015 09:26:28 +0000

As a lot of other part of bash, this is black magic.

Lets suppose a friendly command that spits out NUL separated fields (as find -print0, shyaml get-values-0, ...). Which - may I insist - is the recommended way to communicate wild binary data in a solid way in bash.

How would you parse adequately each individual records by paquets ?

For the purpose of demonstration, lets use the fixed content of a simple data.bin file as our NUL-separated input:

cat < /tmp/data.bin
a:1:b:2 3:c:4
  5:d:6\n7:e::f:9
EOF

Let's verify that we have our NUL bytes:

$ cat /tmp/data.bin | hexdump -v -e '/1 "%02X "'
61 00 31 00 62 00 32 20 33 00 63 00 34 0A 20 20 35 00 64 00 36 5C 6E 37 00 65 00 00 66 00 39 0A

You have noticed that we have some values containing:

spaces (hex: 20),

line breaks (hex: 0A),

a \ followed by a n.

a 0 sized value

a final value 9 ending with a 0a and no final 00.

If using NUL separated fields is recommended, it's to support this kind of data.

I want the implementation of a function read-0 that would allow this type of interaction:

$ cat /tmp/data.bin | while read-0 f1 f2; do
    echo "f1: '$f1', f2: '$f2'"
  done
f1: 'a', f2: '1'
f1: 'b', f2: '2 3'
f1: 'c', f2: '4
  5'
f1: 'd', f2: '6\n7'
f1: 'e', f2: ''
f1: 'f', f2: '9
'

First try

Let's be naive, and we'll use read f1 f2:

$ cat /tmp/data.bin | while read f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
f1: 'a1b2', f2: '3c4'
f1: '5d6n7ef9', f2: ''

You can notice that:

NUL char where ignored for field separation

fields where separated upon consecutive space or return, it uses value stored in IFS environment variable.

their are only 2 lines because the \n was used to separate each record. We should use -d to specify the line delimiter.

Note that the NUL chars are also extracted out of the data as variables don't support the NUL char.

The \ was eaten, because read builtin parse and give it special meaning. We should use -r to avoid that.

But how should we provide the NUL delimiter to the read builtin ? knowing that you can't put NUL chars on the command line ? Hopefully I stumbled onto this blog post: http://transnum.blogspot.sg/2008/11/bashs-read-built-in-supports-0-as.html

Conclusion is that -d '' should be understood magically by bash read builtin to delimit lines with NUL characters.

Better try

Let's apply our new acquired knowledge by trying IFS=$'\0' read -d '' -r f1 f2:

$ cat /tmp/data.bin | while IFS=$'\0' read -d '' -r f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
f1: 'a', f2: ''
f1: '1', f2: ''
f1: 'b', f2: ''
f1: '2 3', f2: ''
f1: 'c', f2: ''
f1: '4
  5', f2: ''
f1: 'd', f2: ''
f1: '6\n7', f2: ''
f1: 'e', f2: ''
f1: '', f2: ''
f1: 'f', f2: ''

That's much better. But notice that:

we didn't get anything in $f2, that's normal: by specifying NUL as line delimiter (with -d '') and having NUL as field delimiter (IFS) we will be doomed to have one field per record. We will need to manage the repacking in a while loop. This doesn't sound too difficult.

where's the final field 0A ? Hum, as there is no NUL final character in the data, read returned errlvl 1 on this last field but filled correctly the variable. A simple echo $f1 prints 9 (if you use this form: while IFS='' read -d '' -r f1 f2; do echo "f1: '$f1', f2: '$f2'"; done < /tmp/data.txt to access variables of the while).

Final Implementation ?

So knowing this, here is the final implemetation of read-0:

read-0() {
    local eof
    eof=
    while [ "$1" -a -z "$eof" ]; do
        IFS='' read -r -d '' "$1" || eof=true
        shift
    done
    test "$eof" != true -o -z "$1"
}

Final ? It surely properly works for our specification test. But what happens if EOF happens before we have fed all the variables ?:

$ echo -n "a" | while read-0 f1 f2; do echo "f1: '$f1', f2: '$f2'"; done
$

Nothing is spit out, despite the fact that we have sent a character.

This is now a specification issue: Do we want read-0 to return errorlevel 0 when it hits EOF while filing the variables ? Okay, but we said 0-sized string was a possible value... read-0 in the current specification knows it hit EOF while filling variables as your first variable can be the 0-sized string. We could make a special case, but I want to be able to distinguish a last empty element from an element.

That read-0, in the actual specification, can't do it. But we can offer a slight change in the way you build your while loop to allow that parsing.

Correct Implementation

To fill partial records, will need another specification change as current implementation will fail whenever it encounters EOF. This is an incompatible specification issue. Aside from this, we need also to take care to actually set the value of the remaining fields to the empty string. This will require to use another version of read-0:

read-0() {
    local eof
    eof=
    while [ "$1" ]; do
        IFS='' read -r -d '' -- "$1" || eof=true
        shift
    done
    test "$eof" != true
}

So this would work with read-0:

$ echo -n a | tr :  '\000' | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: ''

$ echo -n a: | tr :  '\000' | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: ''

Basically, this construct allows a last round in the loop after detecting EOF... and achieve the starting spec:

$ cat /tmp/data.bin | {  eof= ; while [ -z $eof ]; do read-0 f1 f2 || eof=true; echo "f1: '$f1', f2: '$f2'"; done  }
f1: 'a', f2: '1'
f1: 'b', f2: '2 3'
f1: 'c', f2: '4
  5'
f1: 'd', f2: '6\n7'
f1: 'e', f2: ''
f1: 'f', f2: '9
'

Trivial ?

Happy hacking.

bash lore: command substitution and final new lines

vaab — Wed, 07 May 2014 12:38:43 +0000

Be wary that command substitution will remove all final new lines.

Command substitution are $(command ..) construct or older `command ..` one that will be substitued by the standard output of the given command.

But that's incorrect: a proper definition would have been: It will be substitued by the standard output minus any final new lines. For example:

$ value_w_final_dot="$(echo -en "hello\n\n\n.")"
$ value_wo_final_dot="$(echo -en "hello\n\n\n")"
$ echo "My first value: <<$value_w_final_dot>>"
My first value: <>
$ echo "My second value: <<$value_wo_final_dot>>"
My second value: <>

When you are bitten by these types of "features", don't you value more the python motto "explicit is better than implicit" ?.

This chopping occurs at command substitution time. You can definitively store ending new lines in a bash variable, in command parameters, and send them of course over a pipe, just avoid command substitution:

$ value_wo_final_dot="hello"$'\n\n\n'
$ echo "My corrected value: <<$value_wo_final_dot>>"
My first value: <>
$ /bin/echo "As system process argument: " "$value_wo_final_dot"
As system process argument: hello



$

bash lore: NUL character and variables

vaab — Sat, 03 May 2014 10:29:27 +0000

Bash variables are often thought as being able to store any binary content.

Please bear in mind that it can't store NUL character, and only this one.

Here you are:

$ ascii_table() { echo -en "$(echo '\'0{0..3}{0..7}{0..7} | tr -d " ")"; }
$ ascii_table | hd
00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000020  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000030  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000040  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000050  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000060  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000070  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000080  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000090  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000000a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000000b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000000c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000000d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000000e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000000f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000100

But:

$ echo -n "$(ascii_table)" | hd
00000000  01 02 03 04 05 06 07 08  09 0a 0b 0c 0d 0e 0f 10  |................|
00000010  11 12 13 14 15 16 17 18  19 1a 1b 1c 1d 1e 1f 20  |............... |
00000020  21 22 23 24 25 26 27 28  29 2a 2b 2c 2d 2e 2f 30  |!"#$%&'()*+,-./0|
00000030  31 32 33 34 35 36 37 38  39 3a 3b 3c 3d 3e 3f 40  |123456789:;<=>?@|
00000040  41 42 43 44 45 46 47 48  49 4a 4b 4c 4d 4e 4f 50  |ABCDEFGHIJKLMNOP|
00000050  51 52 53 54 55 56 57 58  59 5a 5b 5c 5d 5e 5f 60  |QRSTUVWXYZ[\]^_`|
00000060  61 62 63 64 65 66 67 68  69 6a 6b 6c 6d 6e 6f 70  |abcdefghijklmnop|
00000070  71 72 73 74 75 76 77 78  79 7a 7b 7c 7d 7e 7f 80  |qrstuvwxyz{|}~..|
00000080  81 82 83 84 85 86 87 88  89 8a 8b 8c 8d 8e 8f 90  |................|
00000090  91 92 93 94 95 96 97 98  99 9a 9b 9c 9d 9e 9f a0  |................|
000000a0  a1 a2 a3 a4 a5 a6 a7 a8  a9 aa ab ac ad ae af b0  |................|
000000b0  b1 b2 b3 b4 b5 b6 b7 b8  b9 ba bb bc bd be bf c0  |................|
000000c0  c1 c2 c3 c4 c5 c6 c7 c8  c9 ca cb cc cd ce cf d0  |................|
000000d0  d1 d2 d3 d4 d5 d6 d7 d8  d9 da db dc dd de df e0  |................|
000000e0  e1 e2 e3 e4 e5 e6 e7 e8  e9 ea eb ec ed ee ef f0  |................|
000000f0  f1 f2 f3 f4 f5 f6 f7 f8  f9 fa fb fc fd fe ff     |...............|
000000ff

So bash variables (and $(...) idiom) are reasonably strong: if you know your content hasn't any NUL characters you can safely use them. All this is probably linked to the fact that command line arguments cannot hold NUL characters neither, a common string termination in C programs.

If you really need to store full binary content in a bash variable, you should think about encoding it (base64, xxd, or any format of yours).

Please understand that bash might also be able to do what you want without using variables but only pipes (stdin and stdout). They won't suffer from this limitation of course.

SpamAssassin, Amavis and ubuntu

vaab — Mon, 02 Apr 2012 19:59:01 +0000

If running spamassassin (SA) with amavis without usage of spamd on ubuntu 10.04 and you feel that sa-learn isn't working right, here are a few tips.

I had mails that were false positive and I had to dig a little more than usual to get it working.

learning

Think that SA is executed by amavis user, so:

sudo -u amavis -H sa-learn --showdots {--ham|--spam} YOUR_MAIL_OR_MBOX_OR_MAILDIR

The trick is the sudo part, sa-learn docs can be found everywhere on the net.

checking

This is how to check how spamassassin will rate your mail:

cat badmail | sudo -u amavis -H spamassassin -t

Which is very usefull when used with the same mail, before and after learning.

statistics

Looking at statistics can be usefull to check that the number of spam and ham are consistent with your learning sessions:

sudo -u amavis -H sa-learn --dump magic | grep am

Note

Do not forget to restart amavis with /etc/init.d/amavis restart.

extracting a subpart of a git repository with history

vaab — Wed, 29 Jun 2011 08:10:24 +0000

I recently had to extract a subdirectory of an existing git repository towards its own new repository. And I wanted to keep all the commit information and history that was related to this directory.

EDIT: Even if the solution I used was simpler than I would have expected, I was far from knowing that GIT had a 'filter-branch' command which does the job perfectly well (and it is documented in ProGIT). Thanks Jon for the tip.

sketching the tools used

First step, is to retrieve only the commit Hashes related to this directory:

git log --format=%H -- MY_DIRECTORY

to filter out only the commit Hashes related to this directory. Then:

git format-patch HASH -1

will be used to output patch files with the commit information (subject, author...)

Then, with some sed you'll have to change the directories used in all the outputed patch:

sed -i s%MYOLDDIRECTORY%MYNEWDIRECTORY%g *.patch

The new directory in my case was "" (nothing) as it was in the root folder.

Finally, you'll have to apply your patches to your other repository with:

git am *.patch

Complete commands used

The complete commands that I've used:

cd PARENT_OF_MYDIRECTORY
git log --format=%H -- MYDIRECTORY | tac | \
    (i=0
     while read h; do
         i=$[$i+1]
         git format-patch $h -1 --start-number $i
     done)

This produces NNNN-*.patch files, one for each commit:

sed -i s%MYDIRECTORY%%g *.patch

This will update the paths in all patches:

mv *.patch MY_NEW_REPO
cd MY_NEW_REPO
git am *.patch

A quick check with gitk ensures me that message subject and commit were transfered ok.

Et voilà !

Forcing monospace font in HTML edit post in wordpress

vaab — Sun, 19 Jun 2011 15:00:42 +0000

I'm writing my blog post in ReST code, so naturally I want that the HTML edition tab of wordpress have a fixed-width font as monospace. Here's how to achieve this.

Go edit your own themes file wp-content/themes/mytheme/functions.php, and add these lines:

...
add_action( 'admin_print_styles-post.php', 'my_admin_css' );
add_action( 'admin_print_styles-post-new.php', 'my_admin_css' );
function my_admin_css() {
     ?>
     
     
It's done !
Thanks a lot to WordPress forum...

Using ACL on linux system to mimic samba’s “force user”

vaab — Sat, 21 May 2011 18:05:49 +0000

I'm using samba to share files across a network. I found force user and force group very usefull in a classical team working together on project.

This is the sample /etc/samba/smb.conf:

...
force user = %U
force group = dev-team
create mask = 0775
directory mask = 0775
force create mode = 0660
force directory mode = 0660
...

It ensures that any file created will receive full read, write permission for member of "dev-team" group.

But I have bunch of the user that have a direct access to these directory. And without going through samba, the policy is not enforced.

The solution is quite simple and involves linux ACL.

Install ACL

Your system should already have package acl installed. You can install it (or make sure you already have it) by typing:

apt-get install acl

Then you should edit your /etc/fstab to contain the option acl on the mounted partition you need to get. My fstab looks like:

...
UUID=9cd6deba-e3a6-427c-82e1-48cfaf0793b1 /shares ext4 defaults,acl 0 2
...

Once the /etc/fstab edited, you can remount your partition on the fly with:

mount -o remount /shares

Using ACL

I can mimick the behavior of samba's policy by simply typing:

setfacl -R -d -m u::rwx,g:dev-team:rwx,o::r-x /shares

Options are:

`-R`	is the same than -R for chown or chmod ... it will apply to subdirectory also.
`-d`	is switching to default permissions for newly created files.
`-m`	is adding the following permission rules that are expressed as: u::rwx same as chmod u+rwx g:dev-team:rwx same as chgrp dev-team AND chmod g+rwx o::r-x same as chmod o+rx

You can check ACLs of a given directory with getfacl:

$ getfacl /shares
# file: shares/assistance/
# owner: musicalta
# group: assistance
# flags: -s-
user::rwx
group::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:dev-team:rwx
default:mask::rwx
default:other::r-x

If you need more info on the topic, this blog post was of some help.

http://brunogirin.blogspot.com/2010/03/shared-folders-in-ubuntu-with-setgid.html