Lost Website

So it has it some news outlets that Ubuntu has removed aptitude from its default install. The rationale cited by Ubun-Hero Colin Watson is unsuprising and has been part of what has been used to justify the removal of GIMP:

The base system is constantly tight on space, and everyone wants a piece of that space. I’m afraid we’re going to have to continue to be pretty brutal about what we include there. But, as mentioned, aptitude is of course not being removed from the Ubuntu repository as a whole.

The bug report makes it very clear that aptitude was not removed from the server install. Only the desktop system is affected. The decision of removing the package to obtain some space is simple a cost/gain evaluation from the Ubuntu staff. aptitude isn’t required by the average user that uses what the Ubuntu Desktop user interface offers. It is also important to notice that apt-get is still there.

I’m amazed by the importance some experienced some users have put to the what and what is not included on the Ubuntu install CD. The GIMP is a good exemple of the removal of popular package from the CD. The GIMP is without a doubt the flagship of the open-source world, yet, the people steering Ubuntu have decided to remove it from the default desktop install.

That kind of decision will always make a certain percentage of users unhappy because there is no possible way to justify it in a perfectly objective way. The decision is taken when the cost of removing the package is higher than what could replace it.

In the case of The GIMP, it was decided that more Ubuntu users would benefit from having possibly more package on their default install than there are users that must have The GIMP on the default install. I don’t think anyone can argue that The GIMP benifits all Ubuntu users. It is a heavy application which is too complicated for the tasks most users need to accomplish. They can remove the red eyes of their baby pictures with F-Spot and they can craft their LOLCAT images with a simple paint program.

The rationale for removing aptitude is exactly the same. aptitude is definitely not the program most Ubuntu users should use to install program. Ubuntu developpers have created nice GUI package managers that do the job of any console package manager. Removing a console package manager is also something less for the support team to worry about because no people not caring about it will attempt to use it.

Finally, I think the need for having all sorts of packages offline, on the CD is becoming less than important today. Even people with dialup access to the Internet are able to install packages from the online repository. For the people missing an Internet connection, there are DVDs!

So, to close this article, people unhappy about the removal of aptitude from the main CD should juste do this:

time apt-get install aptitude

An then wonder if the time it took to complain should have been put to better use.

I am the author of bug #501192 in Ubuntu. The bug is about the fact that glib, in Ubuntu, doesn’t use the distribution specific way of getting the default terminal emulator: the x-terminal-emulator link. It’s not a big deal but I could not find any references to it in Launchpad, and elsewhere so I did like what a Good User should do and filed a bug report.

Today the bug went straight out from “New” to “Fix released”. It was never acknowledged, and nobody ever said it was working on it. I have no idea in which distribution the fix was released so I don’t even know if I can use a fixed version on my Karmic box. The bug report page, usually pretty heavy on details on Launchpad, is hopelessly useless.

I can only hope this is a mistake on the part of the reporter. If the bug really has been fixed, then the report really should have more informations about the fix. I don’t file that much bug reports on Ubuntu so I have no idea if this is common practice.

I have asked for more explanations and will this blog up to date.

Here is a welcome digression from my previous Twitter oriented posts. I’m starting to play around with the LD_PRELOAD feature in the Linux dynamic linker. For those who might not know what this feature is, here is the description from ld.so (8).

      LD_PRELOAD
              A whitespace-separated list of additional,  user-specified,  ELF
              shared  libraries  to  be loaded before all others.  This can be
              used  to  selectively  override  functions   in   other   shared
              libraries.   For  setuid/setgid  ELF binaries, only libraries in
              the standard search directories that are  also  setgid  will  be
              loaded.

So in pratical term, any libraries you specify in the LD_PRELOAD environment variable will loaded before any system libraries. This means that dynamic symbols in a loading program will be first searched in those libraries before being searched anywhere else. This means you can override any defined symbol you want in standard libraries.

Let’s start with a rather juvenile example. This will change the behavior of the read (2) function in order to make the user believe a file might have a different content.

ssize_t read(int fd, void *buf, size_t count) {
    static int done = 0;
    if (!done) {
        char silly_str[] = "Haha you got overriden.\n";
        size_t s = count > sizeof(silly_str) ? sizeof(silly_str) : count;
        memcpy(buf, silly_str, s);
        done = 1;
        return s;
    }
    else return 0;
}

If you compile this inside a library that is called, for example, libread.so, you can test this code by running:

> /bin/cat /etc/fstab
# /etc/fstab: static file system information.
#
...
> LD_LIBRARY_PATH=. LD_PRELOAD=libread.so /bin/cat /etc/fstab
Haha you got overriden.

That in itself is just a rather silly prank you can play on your friend’s computer if you happen to have access to it. Experienced programmer will start seeing potential uses for LD_PRELOAD. I am getting to that.

The subject of our next example will be the honorable ls (1). ls uses the opendir (3) function to open a directory and browse its files. It should react properly if it can’t open the directory. One way to test this is to make opendir() return NULL and observe how the caller reacts. You can do that using LD_PRELOAD.

DIR *opendir(const char *name) {
    return NULL;
}

> LD_LIBRARY_PATH=. LD_PRELOAD=libls1.so /bin/ls /tmp
/bin/ls: cannot open directory /tmp

What can you do now if you want to preserve part of the behavior of the function, or modify they result it returns? Your preloaded library will then need to use libdl to dynamically load the function it wants to modify the behavior.

The following example is a very simple override of the opendir (3) function which open a different directory than what the caller expects. I will explain more in detail the details of this function below.

DIR *opendir(const char *name) {
    DIR *(*libc_opendir)(const char *name);
    *(void **)(&libc_opendir) = dlsym(RTLD_NEXT, "opendir");
    return libc_opendir("/tmp");
}

libdl is fortunately very simple to use. The naive approach would be to use dlopen (3) to open the C library, then get the pointer to the function you are calling using dlsym (3). In theory, this technique is valid and working, but doing that circumvents the LD_PRELOAD mechanisme because preloaded libraries can be chained and calling directly into the C library prevents other caller to override our own function.

In practice, calling dlopen() on libc on an Ubuntu Karmic system made some program crash and burn for reasons I will not attempt to explain. The next technique should be preferred on Linux system, especially when dealing with the system C library.

dlsym() has an option that makes the Linux dynamic linker search for the right symbol to be override. This is the RTLD_NEXT flag, which is to be used just for the purpose of wrapper dynamic library functions.

libdl the task of returning the pointer to the right symbol. The RTLD_NEXT option to dlsym() returns the right symbol.

The next and final example of the use of LD_PRELOAD will still use the valiant ls. In time for Christmas, this will modify the output of ls by randomizing the d_type field returned in the dirent structure by readdir (3). If you use colorized ls output, and I believe most of you probably do, you should see a pretty display of color whenever you list a directory by preloading this function.

struct dirent64 *readdir64(DIR *dir) {
    static struct dirent64 *(* libc_readdir64)(DIR *dir) = NULL;
    struct dirent64 *dent;
    unsigned char rnd_dtype[7] = { DT_UNKNOWN, DT_REG,
                                   DT_DIR, DT_FIFO,
                                   DT_SOCK, DT_CHR,
                                   DT_BLK };

    if (libc_readdir64 == NULL) {
        *(void **)(&libc_readdir64) = dlsym(RTLD_NEXT, "readdir64");
        srand(time(NULL));
    }

    dent = libc_readdir64(dir);

    if (dent != NULL)
        dent->d_type = rnd_dtype[rand() % 7];

    return dent;
}

There is still a problem with this code on my new Ubuntu Hardy machine. The code from the preloaded library hangs before the program terminates. I do not understand why this happen and a search for this bug did not turn up anything. The problem doesn’t happen with Ubuntu Karmic.

There is nothing new about using LD_PRELOAD this way. Several very nice libraries have been built with the intention of modifying the behavior of typical libraries.

• fakeroot: “fakeroot provides a fake root environment by means of LD_PRELOAD and SYSV IPC (or TCP) trickery.”
• fakechroot: fakechroot provides a fake chroot environment to programs.
• libtrash:“[...] the shared library which, when preloaded, implements a trash can under GNU/Linux”
• cowdancer: cowdancer is an userland implementation of copy-on-write filesystem.

There are 29 projects matching LD_PRELOAD on freshmeat.net. You might have used some of them.

The code I have written for this demonstration is available on BitBucket.

Adobe Air is a new software platform from Adobe which mixes JavaScript and Flash technologies to enable developers to make rich Internet applications that can run on desktop computers. It is remarkable in the world of proprietary applications in the sense that it has included Linux support early on.

My previous post about Twitter clients might have hinted that I have had bad experience with Air applications in general. In this post I will vent of some of the grudges I have with this new platform on Linux.

This, again, not a very fair review. I have not taken the time to investigate the odds and ends of the platform and will overlook the developer’s point of view on the platform. I have heard over the tubes that programmers working on applications for the Air platform appreciate it, but that’s as far as my investigation (or lack thereof) have taken me.

Air applications aren’t so cross-platform

I have tried 5 Adobe Air applications, mostly Twitter clients: Spaz, TweetDeck, Seesmic Desktop, DestroyTwitter, Tumbleweed, Twhirl. Of those 6 applications only the later 3 worked out of the box on Ubuntu Hardy. The fact cross-platform compatibility isn’t guaranteed by using Air seems to be well-known of developers and most some them will not officially support Linux as an operating platform. This is a very bad average for a technology that is supposed to be cross-platform.

Air applications don’t fail gracefully

The failure mode for each of the non-working applications in also needs to be taken into account when judging quality of the cross-platform Air applications. The behavior of the applications I have tried is less than stellar. TweetDeck is supposed to work on Linux but in the cases where it fails, it shows a semi-helpful error message. Earlier version of TweetDeck failed in the same with Seemic Desktop fails. Seemisc Desktop works partially but the mail display of the application stays empty. SpaZ shows anything usable. Applications failing to work in such a way are very frustrating for the users because they are left to figure out how to use applications that are put in a undetermined state because of holes in the runtime.

Air applications that work on Linux won’t work on all desktops

Adobe Air doesn’t support anything but GNOME and KDE. I usually have the core components of both KDE and GNOME installed on the computers I use, so running applications relying on either desktop environment is not a problem. This is not the case with Adobe Air applications since they apparently rely on some specific features of GNOME or KDE window managers to work properly. This is something that is very seldom seen in the world of X-Window applications. X-Window Applications relying on components of either desktop will usually be happy running on any window manager.

Despite this, Twhirl will work okay-ish in OpenBox with only some graphical glitches to show for.

Adobe Air requires a compositing manager to decent.

I will admit that I’m rather old school and I have not joined the Compiz bandwagon. I did not use any compositing manager on any desktop I use. That means that all Adobe Air applications that use shaped windows, ie: all the ones I have tried, looked awful until I caved-in and enabled the limited compositing feature of KDE 3.5.

Air applications tend focus on the look

Look and feel was never a strong point of Adobe Flash. Flash based application usually don’t try to blend in in the rest of the system. In Adobe Air, Adobe are obviously trying to enforce some Feel in applications, but on the Look side, a lot still leaves to be desired. All Air applications I have tried put a very strong emphasis on design and not so much on blending with the user’s desktop. DestroyTwitter had been designed with fonts a lot smaller than the default size on my desktop. On my CRT screen, it made the application hardly usable. Even the option to use larger font in DestroyTwitter did not do anything to improve the situation.

The worst offender in that category in the applications I have tried is without a doubt Tumblweed, an offline client for the Tumblr web blog system. I had hoped this application would be nicer than editing blog entries through a web site, which is something I dislike. I was mistaken…

The application was removed after a few seconds as it obviously did not focus very much on making editing pleasant.

Okay, I swear this will be my last Twitter inspired post…

In my last post I’ve shown that some people are annoyed at the fact that they can no longer erase their root directory simply by typing rm -rf /. I’m happy that this possibility is removed. I’m not scared that means Linux is being dumbed down. There are already some safeguards in Linux and nobody is complaining about them because they guard even seasoned users to do things that are dangerous or silly.

Proper safeguards

dpkg won’t remove itself

Erasing the primary package manager on your system prevents your from installing any other package. It is hard to argue that it is a good thing to let that happen without warnings.

fdgonthier@moka:~/ > sudo dpkg --purge dpkg
dpkg: error processing dpkg (--purge):
 This is an essential package - it should not be removed.
Errors were encountered while processing:
 dpkg

You can still erase dpkg if you want by using the --force-remove-essential option. I can actually think of reasons somebody might want to do that but this is a bad idea, and probably a mistake, most of the time its attempted. dpkg and apt protect essential and important packages on a Debian system by asking for obscure force options or confirmation, depending on what is attempted.

You can’t unmount /

fdgonthier@moka:~/ > sudo umount /
umount: /: device is busy
umount: /: device is busy

That is akin to removing the wheels of a running car. You can’t even force-unmount it.

You really shouldn’t fsck a drive in use

It is possible to do that, but when you call fsck.ext3 on a mounted partition, it will ask you with a rather serious prompt if you really want to proceed. fsck.ext3 has a force option but it has no effects on this prompt. This speaks volume about how much doing this is discouraged.

In good Unix fashion it is still possible to do it if you really want to but the developer of fsck.ext3 has made clear in the man page that you don’t want to do that.

Some kernel modules can’t be unloaded

modprobe has a force option (-f) too. It can be used to unload reluctant modules. It is clearly documented in modprobe manual page that removing by force may crash your system. I’ve used it a few times, it worked a few times, it crashed a few times, but most of the time it will not work if the module is important.

Why should it be possible to remove the module for the computer chipset while it is running? The answer is left to the imagination of the reader. I think its a good thing this kind of module won’t unload.

You can’t kill the init process

kill -9 1 will do nothing. This is actually hardcoded in the Linux kernel. The init process is the mother of all process in the system. Without it, you won’t be able to boot, or reboot, or use your virtual consoles. It is the reaper of zombies so if it dies, prepare for zombie invasion. There is really no reason to kill this process, ever.

You can’t format a mounted filesystem

fdgonthier@moka:~/ > sudo mkfs.ext3 /dev/sda1
mke2fs 1.41.5 (23-Apr-2009)
/dev/sda1 is mounted; will not make a filesystem here!

mkfs.ext3 offers a way to force this but you have to use the force option (-F)twice.

Files in /dev are dynamically recreated

The health of your system is no longer tied to what is in the mysterious /dev directory. You can delete those files if you wan’t, and, depending what you erase, your currently running system may or may not be affected, but those problems will not survive a reboot or the invocation of /etc/init.d/udev restart.

GUI environment won’t let you in the dark

This is something common now. If for some reason, you change your resolution to something your monitor can’t support and your screen goes blank, desktop environment will automatically switch to your last used resolution. This is a safety mechanism preventing desktop users to set their screen to a resolution that leaves them without graphic display. Any resolution can still be configured statically into the system-wide xorg.conf if necessary.

Conclusion

All the safeguards and behaviors I’ve described above go against the Do What I Mean (DWIM) philosophy of Unix, and the something against the semantic of some option of the command: the kernel won’t let you unload your chipset module because it won’t work without it, mkfs.ext3 won’t let your format a partition that is in use because it’s certainly not what you want to do, dpkg won’t let your remove itself because there is a chance you won’t be able to reinstall it after it was removed.

If you think doing anything of what I’ve named above would be totaly stupid anyway, then you are right. Good safeguards aren’t there to nag you, and won’t ask you to double check everything you do. They simply prevent you from doing things that are possibly catastrophically detrimental to your system. Stop thinking rm -rf / should work because the Unix pilosophy about DYIM, because some command you can type have might no meaning at all.

Then a few things surprised me…

While testing potentially destructive commands in my test virtual machine, I’ve found I was able to do things I think I shouldn’t have. There might be some reason those things are possible. If you know why, please comment.

deluser: delete the root user

You can run deluser root as root without problems. This is of no consequences to the system because the root user is always the user with UID 0, but I think many scripts would be broken by that.

mkfs.ext3: tries to formats a directory

I don’t think a directory can hold a ext3 filesystem, yet mkfs.ext3 /etc will still try to proceed, and fail, after showing a warning telling that will probably won’t work.

fdgonthier@moka:~/ > sudo mkfs.ext3 /etc
[sudo] password for fdgonthier:
mke2fs 1.41.5 (23-Apr-2009)
/etc is not a block special device.
Proceed anyway? (y,n) y
mkfs.ext3: Device size reported to be zero.  Invalid partition specified, or
        partition table wasn't reread after running fdisk, due to
        a modified partition being busy and in use.  You may need to reboot
        to re-read your partition table.

Should it even bother to try?

fdisk: delete an active, mounted partition

fsck.ext3 detects that if targeted partition is mounted. I don’t see why fdisk couldn’t do that. I think there is a good chance that deleting a mounted partition is a mistake and there is not even a warning before proceeding.

unmount /proc and /sys

Those pseudofilesystem have become necessary to a lot of program. It’s a bit weird that you can just unmount them without warnings.

More silly things…

cat /dev/urandom > /dev/mem will very quickly crashes your system. There are multiple variant to that, which are all as efficient as rm -rf / at causing harm to a Linux system.

Those silly commands are unlikely to outside forum of Linux users that love to abuse n00bs, so safeguarding against them is of little importance. Still, why should a superuser shell should have such a raw access to disk devices? Is it necessary? Is there a way to mitigate that?

If you like that post, please subscribe to my
RSS feed. More readers would motivate me to keep writing at least once a week.

A followup…

This entry now has a followup: Good safeguards.

Update

It was pointed to me from Reddit and from a commenter that the -f flag in rm -rf is the force flag and isn’t part of the command. I’m willing to admit I’m wrong and probably overuse the force flag, but I didn’t remember how I got that bad habit.

I then retried the rm -r command on a directory in /tmp. This directory is owned by me but happens to include some files owned by root. I’m allowed to erase those files, but for each file I don’t own, I get a prompt asking me to confirm. This gets annoying quickly and had this not been a test I would have hit Ctrl+C and re-run the command with -f. If you often work with directories with mixed permissions (and I do), you can quicly get into the habit of running rm using the force flag.

This weakens the value of the -f flag for protecting against the huge mistake that is erasing /.

Original post

Did you know that rm -rf / no longer works on recent Ubuntu version? I bet you did not because this command tends to be a bit destructive.

I did not know that. I’ve never been bitten by an accidental rm -rf / but the possibility scares the hell out of me whenever I do a command in my root directory or whenever I write scripts thats erase files.

I’m a seasoned Linux user, yet I was pleased to know that Ubuntu now has that command refuse to work by default. rm -rf / now comes with the implied --preserve-root switch which returns an error when it attemps to erase the root directory.

I’m happy because that command has little reasons to exists beyond the mere fact that rm -rf is meant to unconditionnally erases files and directory: ie, a sole rmsysroot command would not exists. It would make as much sense as a car designed to only crash into the first tree.

Yet, some people oppose such a safeguard. The people that oppose this are probably the kind that made some people come up with sarcasms such as this lovely t-shirt.

LWN’s John Corbet has made a pretty good article about why its a good thing that rm -rf / is safeguarded against errors so I’m not going to rewrite a whole article about that. Let’s just repeat what I find important:

• unguarded rm -rf / might not just happen on the command line, but in scripts too, and at times where you can’t just hit Ctrl+C to save a part of your day
• its just too easy to make a syntax error in a command and type rm -rf / tmp/* when you meant to type rm -rf /tmp/*
• if you really want to remove /, you can use the --no-preserve-root switch.

Now let’s just look at the people who opposes this. Here are some pearls form people complaining about this new feature. I’ve found them on the Launchpad bug report and a blog funnily named Ubuntard. I think naming the authors of those quotes is useless. Suffice to know and weep that they exists.

Also, this directly conflicts with the functionality of the ‘-f’ switch, without which there ALREADY IS CONFIRMATION OF EVERY DELETION.

Bullshit. The -f is not a confirmation, it’s part of the command. Confirmation should come after the command.

So, what’s next? Patch dd to prevent it from wiping your partition?

Well, yes, dd sucks indeed should probably changed. But that’s the subject for another blog post. The point is that I use rm -rf at work up to several times per hour, and I use dd a few times per year. I doubt this is generalizing to say that rm has a much higher potential for errors for every Linux shell users than dd ever has and ever will.

Couldn’t agree more with you. If people don’t know what they are doing, they shouldn’t be working the command line anyway.[...]

rm is not a gun, dynamite, or C4. It erases files. I’ve got 172583 files on this small computer, chances are some of them are superflous and need to be erased. There should not be any remote risk that I erase all my disk if I want to remove just one.

Safety if for bitches. Knowledge is for winners.

No comments. Let’s just say that I hope this person finds a creative, unsafeguarded, way to nuke his system.

[...] come to think of it, if I’d been given a stupid error message telling me not to do that, I’d probably have punched a hole in my CRT instead.

… or yeah, he can do that too! I could go on for quite a while, but the pearl, that is actually repeated more than once, is the invocation of the “Unix philsophy”.

Changes of this nature are destructive to the philosophy of *nix, and is a step closer to the laughable click-the-dialog-box security of Windows Vista.

This is a safeguard; a satefy cushion. This is the program saying “You can’t really be that stupid so I’m not letting you do that!”

Don’t tell me about this so called Unix philosophy! Unix is evolving and when technology evolves, it also tends to grow features that keep people from shooting themselves in the foot. This is why cars have airbags, this is why some laptops have spill guards and free-fall sensors, and also why your water heater doesn’t blow up your house.

Just like people that want to drive without airbags can just remove them, Unix doesn’t bolt the --preserve-root option onto the rm command. It’s probably easy enough to take the coreutils package, change its build options to disable --preserve-root. I could probably write some instructions to do that in a few paragraphs, but since I think doing that would be stupid, I won’t!

This little Bash snippets scans the Debian dpkg database for packages whose configurations files have been modified.

This only touches configurations files which are packaged and made known to dpkg. Some package manually handle their configuration file in maintainer scripts. A good example of that is /etc/network/interfaces which is an important configuration file but not automatically handled by dpkg.

This is as useful as you want to make it. I use it to track changes I’ve done which could end up with a conflict during a package upgrade. If you want a more thorough tracking of configuration changes, use something like etckeeper.

Notice I took some care to use more Bash’isms, to optimize the script. I was able to optimize by just a few percents. The bulk of the time must spent calculating MD5 sums.

#!/bin/bash

for pkg in /var/lib/dpkg/info/*.conffiles; do
    p1=${pkg#/var/lib/dpkg/info/}
    p=${p1%%.conffiles}
    IFS=$'n'
    for cfsum in $(dpkg-query -W -f='${Conffiles}' $p); do
        IFS=' '
        c=($(eval "echo $cfsum"))
        if [ -r ${c[0]} ]; then
            sum=${c[1]}
            cur_sum=($(eval "md5sum ${c[0]}"))
            if [ $sum != ${cur_sum[0]} ]; then
                echo "$p: ${c[0]}"
            fi
        else
            echo "Can't access file: ${c[0]}." > /dev/stderr
        fi
    done
done

This will output something like this:

$ ./check_changed
bash: /etc/bash.bashrc
console-tools: /etc/console-tools/config
libldap2: /etc/ldap/ldap.conf
ntp: /etc/ntp.conf
...

I will not go over the story of the OpenSSL fiasco again. Let’s just say that a developer inserted a particular patch in Debian OpenSSL package that he shouldn’t have. The patch was flawed in ways subtile enough that it escaped review 2 years. Long story short, Debian was left in a rather immense accumulation of fecal matter.

This was bad and made a lot of people object to the fact that software are patched in Linux distributions. It is true that software developers usually are the best person to fix the bugs in their software. What is still true but perhaps less obvious, is that most software developers are not ready or able to work in close cooperation with distribution developers so that their software are perfectly packaged without bugs and in time for a distribution release.

Distribution developers need to patch software for various, good, reasons, to ensure the best quality software distribution to their users, in a timely fashion.

Today I will examine three different kind of patch and try to justify why Debian developers, or distribution developers in general, will develope them. What is very important to note is that Debian, and probably most other distributions, wants that the patch the developers produce are sent to the original developers in the software they are packaging. In the case of the Debian distribution, this is mandated by the Debian Policy Manual, in section 4.3. That way, the patch can be eventually integrated in a future release, and the patch removed for the package for that new release.

The lesser divergence from upstream, the better. The OpenSSL fiasco is certainly a proof of that.

Plain bugfixes

It is a frequent occurance that packages are patched to fix small software bug. It is often inconvenient for a software packager to wait for a new release each time a bug is reported in the software they are responsible of. In some case, that release may come after several months (years), during which the bug will stay unfixed, which may in some case make the package unusable.

Patches are often pulled from the development repository of the software, if they can be backported to the version currently in the distribution.

Debian developers will then patch software when they can, to the best of their knowledge. It is recommended that they consult with the maintainer of the software before uploading their change to the distribution.

Patch example

http://patch-tracking.debian.net/patch/series/view/nano/2.0.7-4/ja_help_freeze.patch

In the case of the nano package, the existance of the patch is documented in the changelog of the package.

Compiler fixes

Like all other package in the distribution, compilers evolve and are eventually upgraded. Debian usually test new compilers ahead of time so they can stamp out compilation bugs early.

In case compilation fails for a package with a new version of a compiler, Debian developers will sometimes prefer to patch the program instead of asking the upstream developer to build his program with a recent or possibly unreleased version of the a compiler. They will patch the program to make it work with the new compiler and transmit the patch to the original developers for their future use.

Patch example

http://patch-tracking.debian.net/patch/series/view/qtodo/0.1.2-5/20-gcc.dpatch

This patch adapts the program to a newer than the one that was probably used to develop the package.

FHS bugfixes

The Debian distribution try to closely follow the Filesystem Hierarchy Standard (FHS). This is actually mandated in the Debian Policy Manual, section 9.1 so, if a program puts a file in a wrong place, it is viewed as a important bug.

Fortunately, most developers will follow the FHS, by habit or consciously. This simplify the packaging process, most program won’t need to be changed to be compliant. Sometimes though, some developers consider that it is not improper that a global configuration file is put outside /etc, or that image files can be put in /var.

Patch example

http://patch-tracking.debian.net/patch/series/view/trac/0.10.3-1etch4/01_use_global_config

This patch tells the Trac web application to look for its configuration file in /etc instead of /usr/share.

That will be enough for today. I think I’m already beyond the “tl;dr” level for most people. I will try to find some more example of types of patch done by packagers in another blog post.

This is an annoyance I found got around to solve.  In Konsole, with a basic X configuration, when you type AltGr Space, you get a non-breakable space. It might happen in other things X terminal/shell combo, or the both of them might not matter at all.

If you are an heavy shell user, you might hit the AltGr Space combo by accident. This wouldn’t be bad if the shell made it somehow obvious that what you entered in a non-breakable space. The fact is that it doesn’t and it’s then totally impossible to distinguish a regular space with a non-breakable space after it was typed in the shell.

cd $HOME
zsh: no such file or directory: cd /home/fdgonthier

After typing such a command quickly, I was often left wondering what the hell was wrong with my command, since, as far as I could see, it was syntaxically correct.

The problem is that since there is a non-breakable space between cd and $HOME, Zsh wants to execute the whole command as a single command. Since I didn’t know I could even type non-breakable space in X, I checked what the hell the shell was doing by using strace. strace is my favorite diagnosis tool for any kind of problems in Linux. Programs don’t lie when they are spied with strace. The problem is obvious:

execve("cd\302\240/home/fdgonthier", ["cd\302\240/home/fdgonthier"], [/* 30 vars */]) = -1 ENOENT (No such file or directory)

It took me a while but I’ve found that \302\240 is the Unicode sequence for non-breakable space. Once I have found that, I Google search lead me to Launchpad bug 218637.

On this page, you see that you can disable non-breakable space using a xkb option in xorg.conf. Simply add the following in the InputDevice section related to your keyboard.

...
Option "XkbOptions"    "nbsp:none"
...

or use

setxkbmap -option "nbsp:none"

in a console, your favorite initialization file, or elsewhere.

The following is a description of the technique I’ve used to create partial Debian mirrors. I have damn near mastered that technique that is not immediately obvious if you don’t know the tool. Nothing there is black magic though, as you will see.

The following example will use the programs called reprepro and
germinate. Both are available on several versions of Ubuntu, Debian Lenny and Debian Sid. I’m not going to explain either in great details. They are well documented in the case of reprepro, and there is enough documentation in germinate.

Required steps

This configuration is necessary in both case.

reprepro is by far my favorite Debian repository manager. It has a steep learning curve at first but once you get past the initial configuration and the first few commands you’ll see it cannot really be made easier.

The first step is to create the directory that will contain the mirror. In that directory create a directory called conf.

In that directory, edit a file named distribution in your favorite editor At this point I usually pop-open the man page for reprepro. Write the following in that file:

Codename: etch
Architectures: amd64 source
Description: Debian Etch (required package only)
Components: main
Update: debian-etch-update

That’s all we need in that file right now. This describes what distribution this mirror will include. You can have several distributions in a mirror. Most field are self explanatory except perhaps the Updates field which will explain below.

Partial mirror, using reprepro only

The next step explain how to create a partial mirror using just reprepro. reprepro is powerful enough to select a subset of a whole package archive.

Name: debian-etch-update
Method: http://gulus.usherbrooke.ca/debian
Components: main
Architectures: amd64 source
FilterFormula: Priority (==required)

The content above is the content of your conf/updates file. It includes all the rules needed to update the distributions configured in the distributions file. The distribution etch we created in is linked to its respective update rule by the Update configuration field.

The key to partial mirroring in this case is the FilterFormula field, which selects just the Debian package in required in a very basic installation. There is also a FilterList field which can select just the set of package you list. This is the basis of the next section.

To start mirroring, go in the mirror directory and do:

reprepro -V update etch

and reprepro will start downloading packages. The -V argument activates verbose mode. After successfully creating the archive, you can reuse the same command to update your mirror.

Germinate and reprepro

germinate is an oddball program which, given a set of package, will recursively find all packages a set of package depend on.

As a simple example, if you tell germinate to germinate the dependencies from the package python, it will list libssl, libreadline and will eventually find libc6. It can also fetch build dependencies if you want everything needed to build the package you want.

We will use germinate to create a Debian repository that mirrors just the set of package the python2.5 package depends on.

germinate is a very specialised tool which sees little use outside making new Debian and Ubuntu derivative distributions. It’s very powerful but a bit confusing. I’ll give the basic recipe you need to know to accomplish our current goal. If there is enough or any interest, I give more use cases for germinate.

1. Create directories called seeds and germinate. The former will include the files we give as input to germinate. The later will include germinate‘s output.

2. In the seeds directory, create files called blacklist, required, STRUCTURE and supported. The blacklist and required will stay empty.

3. In the STRUCTURE file, put the following:

   required:
   supported:
   

4. In the required file, put the following:

    * python2.5
   

   Don’t forget the space at the start of the line.

5. You can then run germinate.

   (MIRROR=http://gulus.usherbrooke.ca/debian
     cd germinate && \
     germinate -v \
               -m $MIRROR \
               -d etch \
               -c main \
               -s seeds \
               -S file:///tmp/temp-distro \
               --no-rdepends)
   

   where you can replace /tmp/temp-distro with the base directory of your mirror.

You should see a bunch of files created in the germinate directory. The only file we are interested in is germinate/required.

germinate output is meant to be human-readable so we need a bit of parsing to extract the required information.

The following Bash magic does the trick:

for pkg in $(cat germinate/required \
  | tail -n +3 \
  | head -n -2 \
  | cut -d '|' -f 1); do\
 echo $pkg install; \
done > mirror.packages

The resulting mirror.package is now suitable to act as a filter for reprepro.

Replace the FilterFormula line in conf/updates by:

FilterList: purge ../mirror.packages

purge tells reprepro not to do anything for packages not in the list.

Then run:

reprepro -V update etch

to create your mirror.