Simple inter-process locks

By: on November 5, 2008

I recently faced a very common problem, how to make sure that only one instance of my program is running at a time on the host.

There are a lot of approaches that can be taken to solve this problem, but I needed a portable solution for Python.

My first idea was to use widely known IPC techniques to lock some global resource. In C I would just create a semaphore and lock it. One problem is that a semaphore is not unlocked when a process dies. Another issue is a lack of support of named semaphores for Python.

The best solution on Unix is to gain an exclusive write lock on a file using fcntl(LOCK_EX).

Of course it doesn’t work on Windows. But for this OS the solution is to take advantage of their mutex facilities using pywin32 module. I was surprised to see that this method works quite well.

It’s also possible to use the fact that only one process at a time can bind to specific tcp/ip port (unless you use SO_REUSEPORT). This is the most portable, but also the most obscure method.

Here’s the code for this inter process “locking”. It’s not really locking, because you can’t block and wait for a lock. All you can do is grab a lock or get an exception. But this is enough to make sure that there is only one process that’s using a resource. This is how you can use this module:

import interlocks, time

lock = interlocks.InterProcessLock("my resource name")
except interlocks.SingleInstanceError:
    print "Other process has acquired this lock."
    print "Press CTRL+C to release the lock..."
    while True: time.sleep(32767)

Test code for the interlocks module needs to open an external process that blocks the resource. The code is not perfect (race conditions), but should be enough for just a test case:

def execute(cmd):
    ''' spawn a new python process that will execute 'cmd' '''
    cmd = '''import time;''' + cmd + '''time.sleep(10);'''
    pid = os.spawnv(os.P_NOWAIT,'/usr/bin/python', ['/usr/bin/python', '-c', cmd])
    time.sleep(1) # poor man's synchronization
    return pid

lock = interlocks.InterProcessLock('test')

# lock resource from other process
pid = execute("import interlocks; a=interlocks.InterProcessLock('test');a.lock();")
try: # fail to grab a lock
except interlocks.SingleInstanceError: print "success: the lock is blocked by spawned process"
else: print "FAILURE: the lock should be blocked by spawned process (pid=%i), but isn't" % (pid,)

os.kill(pid, signal.SIGKILL)
time.sleep(1) # poor man's synchronization

Coding the tests wasn’t so painful, much more problematic was to make tests run on Windows. Obviously we need an os.kill replacement for this platform. The next problem is to make os.spawnv() work on Windows at all: which slashes to use or how to encode spaces in the path. Another issue is that the process pid returned from os.spawnv() can’t be killed. It seems that the return value is not really a proper pid. Don’t waste your time like I did, use subprocess.Popen(). Fixed test code, without os.spawnv is included in the lib.



  1. tef says:


    You should use sys.executable to get the full path of the interpreter, and use to decide to use os.kill on posix, and win32kill on win32 — rather than using hardcoded interpreter paths.

  2. marek says:

    Thanks tef. I just fixed it.

  3. olaf says:

    Nice, but the simplest solution working on all known OS is trying to create a directory. Under Windows, OS/X and UNIX this is guaranteed to be atomic. It fails reliably when the directory exists. This even works in shell scripts!

  4. marek says:

    The problem with directories is that this “resource” is not automatically freed when the owner crashes. You need to use some kind of PID files. Then things became messy. You need to check if the process with this PID exists, if it does one should check if the name is correct and so on.

    That’s why I haven’t written about directories/pid files. All of the resources I wrote about are automatically cleared when the process crashes.

  5. JJ Johns says:

    I found this when looking for a solution to the same problem you describe.
    I had an urge to make this act more like the standard thread lock, so I refactored it a bit and added a locked() method. I also extracted a subclass and made the testing you’d implemented into doctest.
    I also added the ability to have acquire() (aka lock()) take a wait parameter, a ‘la thread locks, for all Lock types.

    Thanks for the code.

  6. staggart says:

    Started to adopt this code for my use. Seemed promising at first until I started testing. I am getting fails in the “release()” (your “unlock” renamed). Seems that somehow that two processes get the lock (perhaps related to a race condition during file unlink?), Anyway, this code will demonstrate it (runs on linux):

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>