This summer Flickr will stop supporting their old authentication system and move fully to OAuth. I've gotten several questions about whether Python FlickrAPI will support that. No worries, it will. If you want it fast, clone the repository, implement OAuth (don't forget the unittests) and send me a pull request.

Python-RSA version 3.0 has been released. A lot of things have been improved:

• Much more secure

• Following industrial standard PKCS#1; this means compatibility with OpenSSL

• Ability to create and verify detached signatures

• More elegant method of encrypting large files

• Ability to save and load keys in PEM and DER format

• Extensive documentation

• More efficient key generation

Check the Python-RSA page for more information!

photo hosted on Flickr

Update: in the mean time 3.0.1 has been released, which removes an import of the "abc" module. This should make the code compatible with Python 2.5 again. All the documentation (including installation instructions) are the same as before.

I'm thrilled to let you know that I've just added encryption and decryption with random padding to my Python RSA module. The code has become much more secure than it was before, and also makes it capable of encrypting and decrypting binary data! This is a big step towards the practicality of the implementation.

I've implemented the padding according to the PKCS#1 standard. The RSA Algorithm page by DI Management explains how to do this very clearly; cudos to them!

Right now these changes are unreleased. I want to add signing and verifying signatures too before I do that. Maybe I'll deprecate the existing functions, as they are much less secure than all the new stuff I'm adding. If you want to try out what I've changed so far, feel free to grab a copy off my Mercurial repository.

photo hosted on Flickr

UPDATE 14:20: I've also implemented PKCS#1 style signatures and signature verification

UPDATE 2011-07-12: the implementation is compatible with OpenSSL, making it very likely compatible with other implementations of PKCS#1 version 1.5!

Back in 2008 I wrote a small post about Python vs. Java. I've revamped it by adding more detail, and tossed in a primer on list comprehensions for free. I hope that once you've read that you're convinced that list comprehensions are a Good Thing.

And as an added bonus, I extended my website so that it can do syntax highlighting!

On to the article!

At my work I'm communicating with an embedded device via a serial-to-USB cable. It works fine, except for one little detail: when the "Ready To Send" (RTS) pin is high, the device shuts down. This is by design, but annoying nonetheless, since almost all serial communication software sets this pin to high. I've written a little Python program that opens the serial connection, sets the RTS pin to low, and shows all data read from it.

You can read the details after the break.

»»» More »»»

A guy named Dave sent me an email today, telling me that my RSA implementation was rather insecure due to the use of the "pickle" Python module. Fortunately I had some nice improvements by Barry Mead already in version control, eagerly waiting to be released. Well, Dave gave me a nice insentive to round up Barry's improvements and send version 2.0 into the world.

Please note that it's a backward-incompatible change. The RSA page shows how you can re-encrypt your data to be compatible (and more secure!) again.

Selenium is a web page testing tool that we use a lot at my work. It can be extended by supplying it with a user-extensions.js file.

We're extending our test application to include support for Selenium Grid. In this setup we can't supply that user-extensions.js file directly, it has to be injected via the client driver. However, this happens too late; Selenium doesn't notice our custom locator functions any more. This is a known issue, also noted by Ross Patterson. I've found a way to work around this limitation. Add the following to your user-extensions.js file such that it's executed after your additions have loaded:

  objectExtend(selenium.browserbot, PageBot.prototype);
selenium.browserbot._registerAllLocatorFunctions();

It seems to work just fine without any unwanted side-effects. If you tried this, please leave a comment to let us know how it worked out for you!

I'm working on a PyQt 4 application for my employer Chess. Of course I'm writing plenty of unit tests for it as well. Those tests have to be cleanly separated from each other, so every test that requires the entire application to be up and running should have its own instance. And this is where it gets tricky, as Qt (and thus PyQt) assumes that there will only be one application, and if that application quits the process quits.

In my unit tests, this is not the case. Every test can potentially start and stop an application, and the test suite will continue running. This caused a segmentation fault (a.k.a. segfault) the second time the QApplication closed.

After a lot of puzzling I found out that QtGui.qApp needs to refer to the running QApplication instance, and be set to None when the application has been closed. After I implemented this the segfaults went away.

This is my ApplicationTest mix-in class that I use for controlling my application lifespan. It contains a little trick to stop the application as soon as it has started:

  class MyApplicationClass(QtGui.QApplication):

  started = QtCore.pyqtSignal()

  def exec_(self):
      self.started.emit()
      return QtGui.QApplication.exec_()

class _ApplicationTest(object):
  '''Mix-in class that can start and stop an application.'''

  APP_CLASS = MyApplicationClass

  def __init__(self):
      self.app = None
      QtGui.qApp = None

  def create_app(self):
      '''Creates and returns a new application'''

      self.app = self.APP_CLASS([])
      QtGui.qApp = self.app
      return self.app

  def stop_app(self):
      '''Stops the application.'''

      if not self.app: return

      self.app.started.connect(self.app.quit,
              type=QtCore.Qt.QueuedConnection)
      self.app.exec_()

      QtGui.qApp = None
      self.app = None

I've been working on a video of our live performance in Winston Kingdom using Adobe AfterEffects. To show the title of a song at the start and end of the song, I used a script to automate the fading. Assign it as an expression to the opacity of the text layer and it'll automatically show for a few seconds at the in-point and out-point of the layer, including nice fades:

  fade_time = 1
show_time = 5
if (time < inPoint + fade_time) {
    linear(time, inPoint, inPoint + fade_time, 0, 100)
} else if (time < inPoint + fade_time + show_time) {
    100
} else if (time < inPoint + fade_time + show_time + fade_time) {
    linear(time, inPoint + fade_time + show_time, inPoint + fade_time + show_time + fade_time, 100, 0)
} else if (time < outPoint - fade_time - show_time - fade_time) {
    0
} else if (time < outPoint - fade_time - show_time) {
    linear(time, outPoint - fade_time - show_time - fade_time, outPoint - fade_time - show_time, 0, 100)
} else if (time < outPoint - fade_time) {
    100
} else if (time < outPoint) {
    linear(time, outPoint - fade_time, outPoint, 100, 0)
} else {
   0
}

fade_time is the duration of fade-in and fade-out, show_time determines how long the text layer is shown at 100% opacity, both are in seconds. You can make an animation preset by adding a "Solid Composite" effect, setting opacity=0% and assigning the expression to the "source opacity" property.

The Python FlickrAPI kit has just been released. The new features are:

• Using auth_callback=False when authentication is actually required now raises a FlickrError exception.

• The implementation uses self.flickr_host so that subclasses can override the API URLs.

• Support for short URLs was added.

Of course the Python FlickrAPI kit is completely dynamic, and so it always supports any new FlickrAPI functionality as soon as it is released by the Flickr team.

The new version can be found at http://stuvel.eu/projects/flickrapi or the Python Package Index at http://pypi.python.org/pypi/flickrapi

Today I released version 1.3 of the Python Flickr API. It's clear that I've been busy with my study and my work, as the world hasn't seen a release in nearly a year. This is mostly due to the dynamic nature of the code - it automatically adapts to any new functionality that's added to Flickr, so no release is required then. This release adds some nifty new things:

• Added functions to easily walk through sets and search results, querying Flickr no more than needed.

• Uses the hashlib module, falling back to the md5 module when hashlib is unavailable.

• Added locking token cache, in case a Flickr API key is used by multiple processes at the same time on the same machine (or shared filesystem)

• Removed the deprecated fail_on_error parameter.

• Implemented the auth_callback functionality.

Go to the Python Flickr API.

I've seen this so many times, and thought about writing about it. Today I found some code that is too good an example to let it go. I'm talking about the refusal some people have to using the continue statement. Here is an example in some pseudo-code:

for value in list {
  if value.condition == True {
      do stuff
      do more stuff
  }
}

It looks like fine code, but personally I'd rather use:

for value in list {
  if value.condition == False: continue

  do stuff
  do more stuff
}

Read more about my coding philosophy after the break

A new version of my pure-Python RSA implementation has been released. The one major change is improved compatibility with Windows.

Download EGG files for Python 2.4, Python 2.5 or the source package at the Python Package index.

Today I released the Python Flickr API version 1.2. Originally the Python interface was made by Beej, and since it's seen many an improvement. Here's what's new in this version:

• Added ElementTree support for Python 2.4.

• Made ElementTree the default response parser.

• The upload and replace methods now take a format parameter.

• Removed deprecated methods.

• Upload and replace methods no longer report progress on their callback regarding the HTTP headers.

Fallout 3 is my latest addiction. It's a great game by the makers of Oblivion. Part of the game is the hacking of computer terminals. It is basically a Master Mind game, but then with letters. You click on a word that shows up in a memory dump of the terminal you are trying to hack, and the terminal tells you how many letters are correct and in the correct place. Of course, a real hacker would write a program to solve the puzzle. So, here it is:

  #!/usr/bin/env python

known = {
    'detector': 1,
}

words = '''
detector carpeted declared detailed dramatic devoured demanded
sinister disposal revealed messages released remained defended
peaceful designed prisoner
'''

words = set(words.split())

def dist(a, b):
    return sum(la == lb for (la, lb) in zip(a, b))

for k in known:
    for w in set(words):
        if dist(k, w) != known[k]:
            words.discard(w)
print "Try any of:", ', '.join(words)

The variable words should be filled with all the words you see on the terminal screen. The known variable should contain a dictionary with the known information you got from the terminal. For example, if you pick those:

• detector: 1/8 correct

• demanded: 2/8 correct

then the known variable should be:

  known = {
    'detector': 1,
    'demanded': 2,
}

If you then run the program (using python fallout.py if you saved the script as fallout.py) it will tell you to try "carpeted" or "messages" as your next guess.

The script requires Python, which is installed by default on most Linux and MacOS systems and should be manually installed on Windows.

update: I have automated the hacking so you can easily use this website to make the sometimes difficult hacking a lot easier.

Sometimes Java just amazes me. As a friend of mine posted, Java does not support Unicode. Of course, my favourite language Python has a much friendlier approach to Unicode:

>>> s = '⿱𠂉乙'.decode('utf-8')
>>> s
u'\u2ff1\U00020089\u4e59'
>>> s[1]
u'\U00020089'
>>> print s[1]
𠂉

Perhaps,. people will some day start to understand that statically typed languages are not a guarantee of a bug-free application. In the end, it's more important that a language does what it is supposed to do.

By popular demand, we've re-licensed our pure-Python RSA module under the European Union Public Licence (EUPL). This new licence is the first open source license to be released by an international governing body.

RSA version 1.2 is code-wise the same as the previous version - the only difference is the new license. It is also still compatible the GPL. Actually, the module is now dual-licensed, under both the GPL and the EUPL. Take your pick, and use which one suits you best.

I've updated this article on 2011-06-15. Comments from back in 2008 are referring to the original version (which cut some corners and was a little too short).

One of the major reasons my company prefers Java over Python, is Java's static type declarations (and all the benefits that follow). If only Python had that, my life would be so much nicer. Here is an example of code I have to work with, and an example of what the Python equivalent would look like.

The background of the code: We have a list of "identifiers" that can identify people who want to use a parking garage. These can be NFC cards, credit cards, and bar codes. We want to filter the list of known identifiers and return only those of type "barcode".

The code in Java:

private List<Identifier> barcodeTypeOnly(List<Identifier> ids)
{
    final List<Identifier> barcodes = new ArrayList<Identifier>();

    for (Identifier id : ids) {
        if (id.getType().equals(Identifier.TYPEBARCODE)) {
            barcodes.add(id);
        }
    }

    return barcodes;
}

What it would look like in Python:

ids = the list of identifiers

barcodes = [identifier for identifier in ids
            if identifier.type == Identifier.TYPEBARCODE]

You could write that last one on a single line, but for clarity I wrote it like this. I understand that the list comprehension construction may not be instantly recognisable, so I wrote a little explanation after the break.

»»» More »»»