Lost Website

I’ve recently bought a 16 GB SDHC card from a Hong Kong seller on eBay. You might think that I was looking for trouble not to buy from a brand name card. Yet, the card was half the price of brand names cards and… it might have worked! I got refunded the card pretty quickly by the seller.

The card was properly detected and could be read by the 2 SD card reader I have at home so I initially thought I had made a good deal. Troubles started to happen when I tried read data I had written on it. The filesystem always ended up in pieces. A bad sector check did not report anything bad.

The final verdict on the quality of the card was determined by an interesting experiment I’ve done.  Here is what I tried.

sudo dcfldd if=/dev/zero of=/dev/sdc

dcfldd is an extension to the venerable dd command. It adds, amongst other features, a transfer status output. This is a welcome improvement from the silent nothing that is running dd. If there is anything you need to remember from this post, it is the existence dcfldd command (in the dcfldd package in Debian and Ubuntu).

Once the card is supposedly filled with zeroes, use the same command to read the content:

sudo dcfldd if=/dev/sdc of=BORKED_SD

Then hexdump (hd) the content of BORKED_SD:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
6abfea50  00 00 00 3f 00 00 00 00  00 00 00 00 00 00 00 00  |...?............|
6abfea60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
74a13000  00 22 00 00 00 06 00 00  00 c7 00 00 00 09 00 00  |."..............|
74a13010  00 00 00 00 00 02 00 00  00 00 00 00 00 0c 00 00  |................|
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
...

Beyond this point, the hexdump data starts to become random.

The hexdump would have only output one line if the card had correctly been filled with zeroes.  The data you see on the next lines should not be there. Either the card returns bad data, or bad data was somehow written on it. You can then understand why the filesystem on the card constantly got corrupted.

Nico’s last blog post touches a subject that has been in my mind for some time now. I must first say that I don’t write this text strictly in reaction to Nico’s post and that I have not verified with if he acknowledges the points I’m about to make.  During the time I’ve spent at Kryptiva it was pretty common to see what I will call WiP (Work-in-progress) commits pushed in our team Mercurial repositories. The reason usually given for pushing broken or incomplete changesets to repositories are the one cited by Nico: people need to backup big changes they are making, or want to complete those changes from another computer.

It would be unacceptable to commit WiP changes on a centralized source control system like SubVersion or CVS because the repository can be checked out by other users at any point in time. Those user tend to expect a working repository even if checking out from a public repository usually means there is a risk that whatever you are checking out will not work. At least, the minimum expectation is that the checked out copy will be compilable.

In distributed version control system (DVCS), like git, everybody commits on it own copy of a repository. Changes get pushed across repositories in discrete bundles. Unless the programming was careless, what ends up in the master repository usually is correct. So, even if programmers have committed broken changes at some point in the repository history, people that clone the repository will usually get a sound copy.

Committing broken code will rarely if ever hurt if all you work on are personal and/or small scale, ashort term, projects. If you are a single programmer tracking changes to a project will git and want to break your trunk every so often, then, go on, be my guess. You are the only person that will suffer your broken history. If you work in a group with several distributed repositories, then you need to read the rest of this post to understand why committing broken trunks is a bad thing.

History

The history of a code repository is the documentation of all the changes that was ever done to a project during its lifetime. As is, it’s the only external documentation that programmer will continuously maintain. This is not something that is obvious when working on projects that have a few tens or maybe hundreds commits. As long as the whole project fits in your head, it is unlikely that you will need to refer to the project change history.  This happens when the project stretches over long time periods and has over thousands of commit. The change history is also something that is very useful when a project changes hand.

WiP commits come into this picture because they usually come with a commit message that not very explicit: “work in progress”, “to be continued”, “I’m not done”, “Finishing tomorrow”, etc. Such a message is extremely not useful if you need to inspect the project history, a blame/annotate log.

In effect, the WiP changesets are separated from the documentation of the change that usually happens at the last commit done on the feature. Tracking back the reason of the change is never unworkable but gets progressively more difficult as the project and the repository age.

Bissection

Bissection is actually a debugging technique that is mostly exclusive to the use of DVCS.  It is a way to find regressions in the repository history by testing past commits using a binary search pattern. At each step of the bissection procedure, the DVCS system updates the repository, putting it in a state represented by a past changeset. The automated bissection procedure then leave the programmeur to test the resulting repository. The programmer should at that point run automated tests or reproduce the problem manually.

This graph represents a set of commit in a repository. The solid lines are connected changesets in the project history. The dashed line represents the changesets touched by the bissection procedure. In this picture, the initial broken changeset is F and the first known good changeset is A. The changeset consulted are, in order, D, B, then C, which is then found to be the changeset that introduced the bug.

This graph illustrates what happens when a few WiP commits are introduced in the tree. WiP commits means the project can’t be compiled at all point in its history which it might be impossible to find a regression using bissection.

This is the most serious problem that can happen if you commit broken code to a repository used by a team. It can seriously hamper debugging in big shared repositories.

To be continued…

If you are not impressed by the 2 reasons I explain here, then you need to read my next post. I think the best reason not to commit broken code is that DVCS offers you all the tools you need to make proper commit. I’ll explain how this is possible with Git and Mercurial in my next post on this subject.

When embedding some window inside Microsoft Outlook, it is not understandable that at some point you need the handle of some an Outlook window object, an Outlook Inspector or an Explorer. The Outlook Object Model does not expose a method to obtain the handle to a window. This is based on some information from Dmitry Streblechenko.

Yes, but not in VBA: you need to QI the Inspector (or Explorer) object for the IOleWindow interface, then call IOleWindow::GetWindow()

If you work with low-level Microsoft Outlook you will eventually find some information, very often forum posts, answered by Dmitry. You will also quickly learn that he is very often right.

I have written the following code in the very last days I have worked on the EchoTracker. It was part of refactorization I have not had time to finish so this code is UNTESTED. This is based solely on my interpretation of the indication of Dmitry.

/// <summary>
/// Embed the Outlook panel in *any* Outlook explorer. Thanks Dimitry.
/// http://www.pcreview.co.uk/forums/thread-1837879.php
/// </summary>
public static MSOWindow GetOutlookWindow(Outlook.Explorer olExp)
{
  IntPtr olExpUnk = IntPtr.Zero;
  IntPtr oleWinPtr = IntPtr.Zero;
  IntPtr hWnd = IntPtr.Zero;
  Guid oleWinGuid = typeof(IOleWindow).GUID;
  IOleWindow oleWin = null;

  try
  {
    olExpUnk = Marshal.GetIUnknownForObject(olExp);
    oleWinPtr = IntPtr.Zero;

    if (Marshal.QueryInterface(olExpUnk, ref oleWinGuid, out oleWinPtr) != 0)
      throw new Exception("QueryInterface failed.");

    oleWin = (IOleWindow)Marshal.GetObjectForIUnknown(oleWinPtr);
    if (oleWin == null)
      throw new Exception("GetObjectForIUnknown failed.");

    oleWin.GetWindow(out oleWinPtr);
  }
  finally
  {
    if (oleWin != null) Marshal.ReleaseComObject(oleWin);
  }
  return new MSOWindow(hWnd);
}

For this code to hopefully work, you need to have the COM interop declaration for the IOleWindow interface. You can find this information on pinvoke.net.

Please, if you stumble upon that code, and happen to have a need for it, use it, or adapt it to your need, leave a comment on this post. I repeat that this is untested. I have no plan to test it, I don’t have a machine on which I can develop Microsoft Outlook.

The Outlook Object Model (OOM) exposes the COMAddins collections of COMAddin object which can be used by Outlook plugins to communicate together. The communication needs to be done through a COM interface. C# and the .NET framework makes it very easy.

You first need to make a ComVisible interface which the caller will use to communicate with the callee. The sample we will work with is a simple class that will call System.Windows.Forms.MessageBox.

using System;
using System.Runtime.InteropServices;

namespace MessageBox
{
    [ComVisible(true)]
    public interface IMessageBox
    {
        void MessageBox(String msg);
    }
}

Next you need to make the callee addin. This addin can be made using VTSO or without it. Addin Express would work too. The callee of course needs to implement IMessageBox interface.

[ComVisible(true)]
[ComDefaultInterface(typeof(IMessageBox))]
public partial class ThisAddIn : IMessageBox
{
    public void MessageBox(String msg)
    {
        System.Windows.Forms.MessageBox.Show("MessageBox() call: " + msg);
    }

    protected override object RequestComAddInAutomationService()
    {
        return this;
    }
    /* ... snip ... */
}

This is not the full code of the addin. I have removed the boring part generated by VTSO.

There are 2 important things to notice in the above code. The first is the ComDefaultInterface attribute, which defines which default interface is exposed to COM by the addin. This is important because the ThisAddin class derives from a non-COM-visible class. The page on the NonCOMVisibleBaseClass Managed Debugging Assistant (MDA) has the information on why this is important.

The next important thing is the implementation of the RequestComAddinAutomationService, which return an instance of the COM class to the caller. This is part of the communication protocol between addins. You can make this method return any instance of a COM visible objects. We’ve used the addin class itself to keep things simple.

Finally, the caller code amounts to accessing the COMAddins collection and finding the right object inside it to get the interface.

object msgBoxID = "MessageBox.Addin";
Office.COMAddIns addins = null;
Office.COMAddIn msgBox = null;
IMessageBox imsgBox = null;

try
{
    addins = Application.COMAddIns;
    msgBox = addins.Item(ref msgBoxID);
    imsgBox = (IMessageBox)msgBox.Object;

    // Actually use the other addin interface.
    imsgBox.MessageBox("Hello, this is a selection change.");
}
finally
{
    if (addins != null) Marshal.ReleaseComObject(addins);
    if (msgBox != null) Marshal.ReleaseComObject(msgBox);
}

This code is pretty straightforward. The 2 important things to notice are the ref object parameter to the COMAddin.Item method and the fact that you need to use the ProgID of the callee addin when searching inside the COMAddins collection. I put emphasis on this because I lost some time trying to find the ProgID of the IMessageBox interface. The ProgID of the addin, when using VTSO, is usually the name of the assembly.

This research was done in the context of the development of the Echotracker although it is still way too early to say what feature will be built on top of that.

This took me a while to figure out and nobody on the Internet seems to have figured out exactly this particular task.

Working on the Echotracker, I wanted to be able to manipulate an array of string from the JScript runtime inside an embedded web browser. Microsoft script runtimes are able to easily manipulate ActiveX objects, and it is easy in C# to make a object visible to COM so interoperability is generally not a problem.

This interoperability wasn’t so intuitive when the time came to deal with arrays. The offending C# declaration was the following.

string[] GetPhoneNumbers();

This is converted by C# into a SAFEARRAY containing BSTR objects. SAFEARRAY is the native name of VB-style multidimensional arrays and BSTR is, in general, the type used by COM to exchange strings. The declaration is nothing exotic as far as COM is concerned. With the MarshalAs attribute added, the declaration now looks like this.

[return: MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VarEnum.VT_BSTR)]
string[] GetPhoneNumbers();

This doesn’t work well with JScript. The first I found thing is that JScript array aren’t compatible with VB array. Microsoft provides a VBArray class for that purpose. It is supposed to deal with SAFEARRAY. Using this class sadly adds some semantic complexity to using arrays returned by ActiveX objects because they can only be used through methods such as getItem(), ubound(), lbound() instead of the normal way for JScript arrays.

Secondly, JScript can’t deal with SAFEARRAY(BSTR) object. Using the GetPhoneNumbers() method crash with the error message “VBArray expected”. The problem is that while JScript can deal with SAFEARRAY, it can’t deal with SAFEARRAY of type other than VARIANT. VARIANT is the catch-all, magic datatype introduced by Microsoft for its dynamic, untyped languages like VB and JScript. This is how I came to test the next declaration.

[return: MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VarEnum.VARIANT)]
string[] GetPhoneNumbers();

This is also not enough for the array to be accepted by JScript, giving out the same “VBArray expected” error message. The problem is in the way the array is marshaled by C#. This page on MSDN clued me to the solution. The .NET marshaller is converting System.String objects automatically into BSTR objects. In this case, the object type needed is a VARIANT object of subtype VT_BSTR. The declaration alone isn’t enough to make the marshaller convert the string to the proper variant type.

[return: MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VarEnum.VARIANT)]
object[] GetPhoneNumbers();

This last declaration makes the .NET marshaller properly convert the strings into VT_BSTR variant object and the array becomes usable by the JScript runtime. In fact, once you know how the .NET runtime converts System.Object arrays, you can do without MarshalAs declaration.

object[] GetPhoneNumbers();

This is, by default, converted into a method returning a SAFEARRAY(VARIANT) object.

The lesson is: leave the .NET COM interop marshaller alone, it probably knows more about COM than you do.

This day has not improved the problems I have with developing on Microsoft Windows. I was doing some research for future development of the Echotracker. I was looking at how to access the Windows address book features through an API and found there are only 3 options open to the programmer wanting to access address book features on Windows.

The main API we use to deal with Microsoft Outlook 200x, the Outlook Object Model (OOM), offers no ways to directly access the address book provided by Outlook. This is bad enough by itself because it means that access to the address book must be done through the dreaded, and unsupported with .NET, MAPI.

Microsoft Windows XP, and probably some earlier version, also had the Windows Address Book API. This is a COM API that give access to the default Windows address book.

This API is deprecated and Microsoft explicitly says that programmers should not use it.

New applications should not use these interfaces. These interfaces exist for backward compatibility with legacy applications. These interfaces will be unavailable in the future.

The newer, Windows Vista and up, alternative is the Windows Contacts API, but here is what Microsoft has to say about this API.

New applications should not use these interfaces. These interfaces exist for backward compatibility with legacy applications. These interfaces will be unavailable in the future.

So Microsoft deprecated the old API, which isn’t necessarily unreasonable, but replaced it with an API that they ask programmers not to use. Of course, they don’t provide any of alternative nor rationale for that message.

So where does that leave us? Nowhere. Programmers who wants to use the Windows address book features will use those APIs, deprecated or not and screw around until it works. Technical analyst taking project decision could also say that Windows has no contact management API. I’m not sure which path Microsoft wants people to take regarding that. I’m lost myself.

As far as I know, it is not that well understood that you can control the Linux firewall (iptables) on a per-user basis, which is something that is sometimes useful on multiuser systems.

Per-user control can be done using the -m owner command line switch of iptables. This matching is of course to be done only on outbound packets put on the OUTPUT chain.

The -m owner match

The owner rule allow to match outgoing packet in several interesting ways. You can of course match on the UID and GID of any user on the system using --uid-owner and --gid-owner. Those 2 arguments match type cover most of the ground you might want to cover in controlling user network access.

The 2 other switch allow you to match on a process ID and a session ID (--pid-owner and --sid-owner). I can see this type of match used inside a daemon launch script to control to which host the process can communicate. Session ID is a lesser known UNIX concept which can match several process launched from a parent. It is good to know those conditions exists but I won’t be discussing them here.

Selectively drop outbound connection

This is something that you may have good reasons to do on a multiuser system. The following rule prevents any outbound connection to any site on port 80, preventing any non-root users from connecting to most of the web.

iptables -A OUTPUT -p tcp --dport 80 -m owner ! --uid-owner 0 -j DROP

You can of course use --gid-owner to make a more sensible control using a group.

Prevent incoming and outgoing connections

Eventhough you can’t select which user can receive inbound connection request, you can still prevent a specific user process from accepting connection from the outside. You can do that by dropping packets that take part in the TCP connection handshake.

The TCP connection handshake is a process that begin when a connection is attempted on a service port. The first packet sent in that handshake is a packet that has a special flag, called SYN, raised. If the service port can answer to that connection request, it needs to send a TCP packet with 2 flags, SYN and ACK, raised. In practice, this means that if packets that have the SYN and ACK flag raised are blocked, incoming connections will never succeed.

iptables -A OUTPUT -p tcp --tcp-flags ALL SYN,ACK -m owner ! --uid-owner -j REJECT

This blocks the completion of connections attempts done on all local port by non-root users. You could as well selectively ACCEPT connections only to the well known service on your multiuser machine but you would have to remember to modify your firewall script every time you enable another service. You can also selectively allow certain class of user to receive connections using --gid-owner.

I can’t guarantee the level of security offered by the rules I propose here. There are other alternative than using firewalls to control network security on multiuser system. My last hosting provider HCOOP is using grsec which is a set of patch over the Linux kernel.

I am the author of bug #501192 in Ubuntu. The bug is about the fact that glib, in Ubuntu, doesn’t use the distribution specific way of getting the default terminal emulator: the x-terminal-emulator link. It’s not a big deal but I could not find any references to it in Launchpad, and elsewhere so I did like what a Good User should do and filed a bug report.

Today the bug went straight out from “New” to “Fix released”. It was never acknowledged, and nobody ever said it was working on it. I have no idea in which distribution the fix was released so I don’t even know if I can use a fixed version on my Karmic box. The bug report page, usually pretty heavy on details on Launchpad, is hopelessly useless.

I can only hope this is a mistake on the part of the reporter. If the bug really has been fixed, then the report really should have more informations about the fix. I don’t file that much bug reports on Ubuntu so I have no idea if this is common practice.

I have asked for more explanations and will this blog up to date.

COM stands for Component Object Model. This is the technology behind most of non-trivial things in Windows, the why of the obscure HKEY_CLASSES_ROOT registry subtree and the core of the infamous ActiveX technology.

Now, I don’t know much about COM programming so this won’t be a tutorial on the subject. I know just the minimum I need to safely work with MAPI in the context of what I do at Kryptiva.

COM programming include many concepts of typical object-oriented programming languages. It’s just a lot more verbose. You can view COM interfaces as structures containing pointers to functions. COM classes which implemented those interfaces are managed using reference counting. This means that whenever you carry a reference to a COM class around you need to increment the reference count manually and release it when you are done.

This is something tedious and error-prone but unavoidable when you are using COM objects in C. Luckily, C++.NET isn’t C and it means we can use RAII, just like the code I presented in Naivety in C++.

Before

Before presenting the simple wrapper, let’s see how plain COM programming would look like without the wrapper.

IUnknown *pUnk;
IMAPIProp *pprop;
HRESULT r;

pUnk = (IUnknown*)Marshal::GetIUnknownForObject(mapiObj).ToPointer();

r = pUnk->QueryInterface(::IID_IMAPIProp, (void **)&pprop);
if (FAILED(r))
  Marshal::ThrowExceptionForHR(r);

r = pProp->SaveChanges(FORCE_SAVE | KEEP_OPEN_READWRITE);
if (FAILED(r))
  throw gcnew MapiException(pProp, r, "Failed to save attachment data");

pUnk->Release();
pProp->Release();

This is simple MAPI code that I hacked as a demonstration. It doesn’t do anything interesting.

This code is pretty simple but gives an idea about how hairy code that is calling COM can get, especially if you throw along exception handling.

After

Here is the same code, using the simple wrapper that is listed below.

COM<IMAPIProp> ifProp;
COM<IUnknown> ifUnk;
HRESULT r;

pUnk.Ptr = static_cast<IUnknown *>(Marshal::GetIUnknownForObject(mapiObj).ToPointer());

r = ifUnk.Ptr->QueryInterface(::IID_IMAPIProp, reinterpret_cast<void**>(&ifSession.Ptr))
if (FAILED(r))
   Marshal::ThrowExceptionForHR(r);

r = pProp->SaveChanges(FORCE_SAVE | KEEP_OPEN_READWRITE);
if (FAILED(r))
   throw gcnew MapiException(pProp, r, "Failed to save attachment data");

You can see that the COM wrapper releases the COM objects which is a precious feature as the calling procedure gets more complete. If the end result is a bit anticlimatic, it is because error handling is still in the way. The wrapper I have made for error handling is a lot less pretty so I won’t be presenting it here.

The code itself

// RAII friendly wrapper over COM interface usage.
template <class C> class COM
{
public:
   C *Ptr;

COM<C>(C *pCom)
{
   assert(pCom);
   Ptr = pCom;
}

COM<C>()
{
   Ptr = 0;
}

COM<C>& operator=(const COM<C>& com)
{
   assert(com.Ptr != 0);
   Ptr = com.Ptr;
   Ptr->AddRef();
   return *this;
}

COM<C>(const COM<C>& com)
{
   assert(com.Ptr != 0);
   Ptr = com.Ptr;
   assert(Ptr->AddRef() >= 1);
}

~COM<C>()
{
   if (Ptr) assert(Ptr->Release() >= 0);
   Ptr = 0;
}
};

There is not much to say about this code. It’s very simple and does the job. The copy constructor is there to increase the reference count in case an instanciated COM object is passed between callers and callee.

I have added assert calls to catch potential problems I may have missed but none of them have ever been triggered.

I’m very bad at keeping track of the time I spend working. This tends to require manual input, and something to remind me of doing the input. The later part is where I usually fail and lose interest. This meant that last week I had to input a year worth of timekeeping data in a few hours in a web application for that purpose.

This is not a problem as opaque as it might seem to some people. We use timekeeping at work to keep track of how much time are spent doing specific projects and not to keep a precise account of who is working or not at specific time.

The only place where that data is consigned is in out revision control systems, Mercurial. It has a detailed log of the data that was commited inside a repository and, an explanation why if the commit message was good. Scanning each repository (all 72 of them) with the default log command output would have been undoable.

Luckily, Mercurial has a lesser known feature which allows users to present log data data in a more terse way that the default. This is the --template switch, which is pretty well explained in Mercurial manual.

The command I’m using in the script bellow is something like that:

hg log --template "{date|shortdate} {author|email} {rev}"

Here is an excerpt of the output of this command.

...
2009-09-09  fdgonthier@kryptiva.com 1934
2009-09-21  fdgonthier@kryptiva.com 1935
2009-09-21  fdgonthier@kryptiva.com 1936
...

So this shows some commit I have done in a specific project during the month of september in 2009. It was then trivial to extract that data from all the repositories to see what I was working on at what date. The following script loops around all my repositories and extract from the log the dates in 2009 where I have commited something. Note that I have added another field in the template, which is the name of the directory containing the Mercurial repository. This will be used to distinguish between projects in the step after the data is obtained.

#!/bin/sh

for i in $(find . -maxdepth 1 -type d | cut -c 3-); do
  if [ -e $i/.hg ]; then
    echo "Churning $i"
    (cd $i; \
       hg log \
         --template "{date|shortdate}  $i {author|email} {rev}\n" |\
      grep -E "^2009.*(fdgonthier)") > ~/churn/$i
  fi
done

From the files churn directory it’s then trivial to get a picture of everything that was worked on all through the year. Just cat the file together and sort the whole set of lines by date.

> cd ~/churn && cat * | sort | less
...
2009-03-03  bar-daemon fdgonthier@kryptiva.com 1803
2009-03-03  bar-daemon fdgonthier@kryptiva.com 1804
2009-03-04  libfoo fdgonthier@kryptiva.com 5
2009-03-04  libfoo fdgonthier@kryptiva.com 6
2009-03-04  bar-daemon fdgonthier@kryptiva.com 1805
2009-03-04  bar-deamon fdgonthier@kryptiva.com 1806
...

This will be as accurate as you keep your repositories clean. For example, it might be difficult to extract only the changesets you did if you did not pay attention to correctly configuring your default commit name. It happened to me in some contexts. I also had to use the revision number of the log to the content of some commits because I could not remember to what subproject they were attached.

This is not something you want to have to do. It’s much more accurate and easy to properly feed the timetracking program on a daily basis. There is no excuse not do to it properly, but if you tend to forget that kind of thing, this trick can help.