Interfacing multi-threaded C API's with Python using SWIG ...

Ran into an interesting problem yesterday. There is a third party C library that I need to interface to using Python. Normally, this isn't a problem. Just use SWIG to make a Python Wrapper on the desired API functions and call away. But this library was a little nastier because it spawned other threads and notified the caller via callbacks. Callbacks on their own aren't really a big deal for SWIG, you just have to handle the delegation from C to Python yourself, which is easy to do in the .i file. But when the callback is coming from another thread there are some twists.

CPython is notorious for the Global Interpreter Lock (GIL). You have to release the GIL whenever you do blocking I/O and re-obtain it when you are done. With a callback it's a little different, you want to acquire the lock, call Python and release it again when your back in C-land.

Starting off, my .i file looked something like this:

%module probe
%{
#include "myapi.h"
%}

%include "windows.i"

typedef void __stdcall (*HandleDataCallBack)(char*, int);

int Start(HandleDataCallBack);
int Stop();
int SendInquiry();

%{
PyObject* python_callback = 0;

void __stdcall api_start_callback(char* foo, int blah)
{
PyObject *arglist;
PyObject *result;
PyGILState_STATE gstate;

if (!PyCallable_Check(python_callback))
{
PyErr_SetString(PyExc_TypeError, "Python callback must be callable");
return;
}
arglist = Py_BuildValue("(si)", foo, blah);

gstate = PyGILState_Ensure();

result = PyEval_CallObject(python_callback, arglist);

Py_XDECREF(arglist);
Py_XDECREF(result);

PyGILState_Release(gstate);
}

void alt_start(PyObject *pyfunc)
{
python_callback = pyfunc;
Start(api_start_callback);
Py_INCREF(pyfunc);
}
%}

void alt_start(PyObject*);

Essentially what I'm doing here is defining a new function alt_start() to replace the API Start() function. A well known callback (api_start_callback) is then handed off the the real Start(). api_start_callback() does the heavy lifting of assembling the arguments, acquiring the GIL, calling the Python callback and then giving the GIL back to Python.

For other places where the threading is an issue, SWIG can take of this itself when you run Swig with the -threads option. In my case: swig -python -threads probe.i

The simplified Python code for calling this looks something like this:

import probe
import time

def ProbeCallback(foo, blah):
....print "Foo: %s, Blah: %d" % (foo, blah)

probe.alt_start(ProbeCallback)
probe.SendInquiry()
time.sleep(60)
probe.Stop()

Someday perhaps, I'll be able to call Start() directly and pass my Python callback handler without the need for this special handling code. Until then, the work around isn't too horrible.

Microsoft VC++ SxS Assemblies & Manifests

This is true madness. In the process of bringing a legacy VC++ project up to date with Visual Studio 2008 there were some real hurdles to get over.

The biggest issue is that starting with Visual Studio 2005 Microsoft introduced the idea of Side-by-Side Assemblies and Manifests. This is a mechanism whereby different versions of the Visual Studio C runtime, ATL libraries, MFC libraries and other assemblies can be run "side-by-side".

So, while your program may compile and run in debug, it may not be able to be copied to a system with no dev environment on it without serious tap-dancing. You'll get cryptic "The application is not configured correctly and should be reinstalled" messages on XP or a vague hint about SxS on Vista.

After a lot of time spent on the MSDN forums I finally figured out how this mess works.

First off, VC++ apps will include \Program Files\Microsoft Visual Studio 9.0\VC\include\crtdefs.h via Stdafx.h. Look at this header file. It contains lines like this:

#include <crtassem.h>

#ifdef _M_IX86
#ifdef _DEBUG

#pragma comment(linker,"/manifestdependency:\"type='win32' " \
      "name='" __LIBRARIES_ASSEMBLY_NAME_PREFIX ".DebugCRT' "         \
      "version='" _CRT_ASSEMBLY_VERSION "' "                          \
      "processorArchitecture='x86' "                                  \
      "publicKeyToken='" _VC_ASSEMBLY_PUBLICKEYTOKEN "'\"")
#else

#pragma comment(linker,"/manifestdependency:\"type='win32' "            \
      "name='" __LIBRARIES_ASSEMBLY_NAME_PREFIX ".CRT' "              \
      "version='" _CRT_ASSEMBLY_VERSION "' "                          \
      "processorArchitecture='x86' "                                  \
      "publicKeyToken='" _VC_ASSEMBLY_PUBLICKEYTOKEN "'\"")
#endif

#endif /* _M_IX86 */

This will put a little piece of code in the .obj which gets picked up by the linker and the manifest tool. So, if _DEBUG if defined anywhere, you'll get manifest entries requiring the debug runtimes. So, double/triple check that you don't have any debug files or .libs in your project. The .manifest file looks like this:

<assembly xmlns="'urn:schemas-microsoft-com:asm.v1'" manifestversion="'1.0'">
<trustinfo xmlns="urn:schemas-microsoft-com:asm.v3">
  <security>
    <requestedprivileges>
      <requestedexecutionlevel level="'asInvoker'" uiaccess="'false'">
      </requestedexecutionlevel>
    </requestedprivileges>  
  </security>
</trustinfo>
<dependency>
  <dependentassembly>
    <assemblyidentity type="'win32'" name="'Microsoft.VC90.CRT'" version="'9.0.30729.4148'" processorarchitecture="'x86'" publickeytoken="'1fc8b3b9a1e18e3b'">
     </assemblyidentity>
  </dependentassembly>
</dependency>
<dependency>
  <dependentassembly>
    <assemblyidentity type="'win32'" name="'Microsoft.VC90.MFC'" version="'9.0.30729.4148'" processorarchitecture="'x86'" publickeytoken="'1fc8b3b9a1e18e3b'">
  </assemblyidentity>
  </dependentassembly>
</dependency>
</assembly>

The big thing to note is the version of the runtime. How does Visual Studio know which version of the runtime to load? Well, it uses \Program Files\Microsoft Visual Studio 9.0\VC\include\crtassess.h to make this decision. Whenever you install a new Service Pack for Visual Studio this file gets updated.

If you define _BIND_TO_CURRENT_VCLIBS_VERSION = 1 in your C++ code, crtassess.h will grab the latest MFC, ATL and CRT libraries, whatever they are. These are the versions that will go into the .manifest file (by the way, set up your code to embed the .manifest in your code to keep things easier).

On the deployment machine, you need to install all the Visual Studio runtime libraries. The best way to do this is by running the latest and greatest vcredist_x86.exe application from the microsoft site. But this has its own problems.

The entire version number of the assembly is important. With VS 2008 SP1, the CRT version was bumped from 9.0.21022.8 to 9.0.30729.4148, but the redist for SP1 installed 9.0.30729.1 and the program wouldn't run on the deployment machine. Turns out there was a Security Update for SP1 and a later redist program for it. I don't know how you're supposed to keep up on this other than watch the msdn forums. Once this was run on the deployment machine the program would run.

Another problem I ran into was the compiler kept putting entries in the manifest for the older 9.0.21022.8 crt even though I had _BIND_TO_CURRENT_VCLIBS_VERSION defined everywhere. The workaround for this is very cryptic. You need to put the following code somewhere in your application (like in stdafx.h)

extern "C" { 
__declspec(selectany)       int _forceCRTManifestRTM;
__declspec(selectany)       int _forceCRTManifestCUR;
}; 

This will force the compiler to use the latest assemblies.

You can see how the manifests are looked up on your deployment machines by looking in \Windows\WinSxs. You'll see the mapping of the runtime to the .dll's along with the security policies for selecting them. Using this you can quickly spot any discrepancies between your development environment and your deployment environment.

Some Changes ...

I've nuked my old blog. Since all my family photos and ramblings are on Facebook I'm going to have this site focus on public industry related topics. Stay tuned.