This summer Flickr will stop supporting their old authentication system and move fully to OAuth. I've gotten several questions about whether Python FlickrAPI will support that. No worries, it will. If you want it fast, clone the repository, implement OAuth (don't forget the unittests) and send me a pull request.

Python-RSA version 3.0 has been released. A lot of things have been improved:

• Much more secure

• Following industrial standard PKCS#1; this means compatibility with OpenSSL

• Ability to create and verify detached signatures

• More elegant method of encrypting large files

• Ability to save and load keys in PEM and DER format

• Extensive documentation

• More efficient key generation

Check the Python-RSA page for more information!

photo hosted on Flickr

Update: in the mean time 3.0.1 has been released, which removes an import of the "abc" module. This should make the code compatible with Python 2.5 again. All the documentation (including installation instructions) are the same as before.

I'm thrilled to let you know that I've just added encryption and decryption with random padding to my Python RSA module. The code has become much more secure than it was before, and also makes it capable of encrypting and decrypting binary data! This is a big step towards the practicality of the implementation.

I've implemented the padding according to the PKCS#1 standard. The RSA Algorithm page by DI Management explains how to do this very clearly; cudos to them!

Right now these changes are unreleased. I want to add signing and verifying signatures too before I do that. Maybe I'll deprecate the existing functions, as they are much less secure than all the new stuff I'm adding. If you want to try out what I've changed so far, feel free to grab a copy off my Mercurial repository.

photo hosted on Flickr

UPDATE 14:20: I've also implemented PKCS#1 style signatures and signature verification

UPDATE 2011-07-12: the implementation is compatible with OpenSSL, making it very likely compatible with other implementations of PKCS#1 version 1.5!

Back in 2008 I wrote a small post about Python vs. Java. I've revamped it by adding more detail, and tossed in a primer on list comprehensions for free. I hope that once you've read that you're convinced that list comprehensions are a Good Thing.

And as an added bonus, I extended my website so that it can do syntax highlighting!

On to the article!

At my work I'm communicating with an embedded device via a serial-to-USB cable. It works fine, except for one little detail: when the "Ready To Send" (RTS) pin is high, the device shuts down. This is by design, but annoying nonetheless, since almost all serial communication software sets this pin to high. I've written a little Python program that opens the serial connection, sets the RTS pin to low, and shows all data read from it.

You can read the details after the break.

»»» More »»»

A guy named Dave sent me an email today, telling me that my RSA implementation was rather insecure due to the use of the "pickle" Python module. Fortunately I had some nice improvements by Barry Mead already in version control, eagerly waiting to be released. Well, Dave gave me a nice insentive to round up Barry's improvements and send version 2.0 into the world.

Please note that it's a backward-incompatible change. The RSA page shows how you can re-encrypt your data to be compatible (and more secure!) again.

I'm working on a PyQt 4 application for my employer Chess. Of course I'm writing plenty of unit tests for it as well. Those tests have to be cleanly separated from each other, so every test that requires the entire application to be up and running should have its own instance. And this is where it gets tricky, as Qt (and thus PyQt) assumes that there will only be one application, and if that application quits the process quits.

In my unit tests, this is not the case. Every test can potentially start and stop an application, and the test suite will continue running. This caused a segmentation fault (a.k.a. segfault) the second time the QApplication closed.

After a lot of puzzling I found out that QtGui.qApp needs to refer to the running QApplication instance, and be set to None when the application has been closed. After I implemented this the segfaults went away.

This is my ApplicationTest mix-in class that I use for controlling my application lifespan. It contains a little trick to stop the application as soon as it has started:

  class MyApplicationClass(QtGui.QApplication):

  started = QtCore.pyqtSignal()

  def exec_(self):
      self.started.emit()
      return QtGui.QApplication.exec_()

class _ApplicationTest(object):
  '''Mix-in class that can start and stop an application.'''

  APP_CLASS = MyApplicationClass

  def __init__(self):
      self.app = None
      QtGui.qApp = None

  def create_app(self):
      '''Creates and returns a new application'''

      self.app = self.APP_CLASS([])
      QtGui.qApp = self.app
      return self.app

  def stop_app(self):
      '''Stops the application.'''

      if not self.app: return

      self.app.started.connect(self.app.quit,
              type=QtCore.Qt.QueuedConnection)
      self.app.exec_()

      QtGui.qApp = None
      self.app = None

The Python FlickrAPI kit has just been released. The new features are:

• Using auth_callback=False when authentication is actually required now raises a FlickrError exception.

• The implementation uses self.flickr_host so that subclasses can override the API URLs.

• Support for short URLs was added.

Of course the Python FlickrAPI kit is completely dynamic, and so it always supports any new FlickrAPI functionality as soon as it is released by the Flickr team.

The new version can be found at http://stuvel.eu/projects/flickrapi or the Python Package Index at http://pypi.python.org/pypi/flickrapi

Today I released version 1.3 of the Python Flickr API. It's clear that I've been busy with my study and my work, as the world hasn't seen a release in nearly a year. This is mostly due to the dynamic nature of the code - it automatically adapts to any new functionality that's added to Flickr, so no release is required then. This release adds some nifty new things:

• Added functions to easily walk through sets and search results, querying Flickr no more than needed.

• Uses the hashlib module, falling back to the md5 module when hashlib is unavailable.

• Added locking token cache, in case a Flickr API key is used by multiple processes at the same time on the same machine (or shared filesystem)

• Removed the deprecated fail_on_error parameter.

• Implemented the auth_callback functionality.

Go to the Python Flickr API.

To sync my mail between computers I use offlineimap on a secure filesystem. Today I mistakenly ran offlineimap before mounting the secure filesystem, which caused it to duplicate all emails. Not wanting to do any manual work to fix this, I wrote a small Python 3 program that repaired the damage.

Read on after the break for the code.

»»» More »»»

I've released a new version of Multiblend! Version 1.4 has the following improvements over the previous version:

• Display timing info after rendering.

• Added -E option to skip existing frames and render only the missing ones.

• Smarter distribution of frames.

• Implemented simpler logging.

• Verifies that the output directory exists before rendering.

• Added 'cachesync' executable that can distribute the Blender cache directory (such as used with particle systems) to nodes.

• Removed use of the md5 module when using Python 2.6 or newer.

• Replaced the popen2 module with the subprocess module. This limits the compatibility of Multiblend to Python 2.4 or newer on the master node.

• Split the implementation into several modules to increase maintainability and insulation of the parts.

Multiblend connects to multiple computers on your network, and lets Blender render in parallel on each. This will drastically improve your rendering speeds!

A new version of my pure-Python RSA implementation has been released. The one major change is improved compatibility with Windows.

Download EGG files for Python 2.4, Python 2.5 or the source package at the Python Package index.

Today I released the Python Flickr API version 1.2. Originally the Python interface was made by Beej, and since it's seen many an improvement. Here's what's new in this version:

• Added ElementTree support for Python 2.4.

• Made ElementTree the default response parser.

• The upload and replace methods now take a format parameter.

• Removed deprecated methods.

• Upload and replace methods no longer report progress on their callback regarding the HTTP headers.

Fallout 3 is my latest addiction. It's a great game by the makers of Oblivion. Part of the game is the hacking of computer terminals. It is basically a Master Mind game, but then with letters. You click on a word that shows up in a memory dump of the terminal you are trying to hack, and the terminal tells you how many letters are correct and in the correct place. Of course, a real hacker would write a program to solve the puzzle. So, here it is:

  #!/usr/bin/env python

known = {
    'detector': 1,
}

words = '''
detector carpeted declared detailed dramatic devoured demanded
sinister disposal revealed messages released remained defended
peaceful designed prisoner
'''

words = set(words.split())

def dist(a, b):
    return sum(la == lb for (la, lb) in zip(a, b))

for k in known:
    for w in set(words):
        if dist(k, w) != known[k]:
            words.discard(w)
print "Try any of:", ', '.join(words)

The variable words should be filled with all the words you see on the terminal screen. The known variable should contain a dictionary with the known information you got from the terminal. For example, if you pick those:

• detector: 1/8 correct

• demanded: 2/8 correct

then the known variable should be:

  known = {
    'detector': 1,
    'demanded': 2,
}

If you then run the program (using python fallout.py if you saved the script as fallout.py) it will tell you to try "carpeted" or "messages" as your next guess.

The script requires Python, which is installed by default on most Linux and MacOS systems and should be manually installed on Windows.

update: I have automated the hacking so you can easily use this website to make the sometimes difficult hacking a lot easier.

Sometimes Java just amazes me. As a friend of mine posted, Java does not support Unicode. Of course, my favourite language Python has a much friendlier approach to Unicode:

>>> s = '⿱𠂉乙'.decode('utf-8')
>>> s
u'\u2ff1\U00020089\u4e59'
>>> s[1]
u'\U00020089'
>>> print s[1]
𠂉

Perhaps,. people will some day start to understand that statically typed languages are not a guarantee of a bug-free application. In the end, it's more important that a language does what it is supposed to do.

By popular demand, we've re-licensed our pure-Python RSA module under the European Union Public Licence (EUPL). This new licence is the first open source license to be released by an international governing body.

RSA version 1.2 is code-wise the same as the previous version - the only difference is the new license. It is also still compatible the GPL. Actually, the module is now dual-licensed, under both the GPL and the EUPL. Take your pick, and use which one suits you best.

I've updated this article on 2011-06-15. Comments from back in 2008 are referring to the original version (which cut some corners and was a little too short).

One of the major reasons my company prefers Java over Python, is Java's static type declarations (and all the benefits that follow). If only Python had that, my life would be so much nicer. Here is an example of code I have to work with, and an example of what the Python equivalent would look like.

The background of the code: We have a list of "identifiers" that can identify people who want to use a parking garage. These can be NFC cards, credit cards, and bar codes. We want to filter the list of known identifiers and return only those of type "barcode".

The code in Java:

private List<Identifier> barcodeTypeOnly(List<Identifier> ids)
{
    final List<Identifier> barcodes = new ArrayList<Identifier>();

    for (Identifier id : ids) {
        if (id.getType().equals(Identifier.TYPEBARCODE)) {
            barcodes.add(id);
        }
    }

    return barcodes;
}

What it would look like in Python:

ids = the list of identifiers

barcodes = [identifier for identifier in ids
            if identifier.type == Identifier.TYPEBARCODE]

You could write that last one on a single line, but for clarity I wrote it like this. I understand that the list comprehension construction may not be instantly recognisable, so I wrote a little explanation after the break.

»»» More »»»