• ViM's OmniComplete is actually pretty decent. Calltip support in the GUI is also good. (GUI? ViM? Yeah, weird.)
• Emacs completion, from Rope, is also good. Emacs's refusal to make any concession to GUIs though keeps things clunky. Not that it isn't great that Everything Works over plain ssh; that's fine, but going through classic Emacs buffers for docstrings or completion means everything takes more keystrokes than it should while being less useful than having that information Always On.
• Rope also gives Emacs refactoring support that works surprisingly well.
• PyDev still sees a big win from the Eclipse platform. Specifically, even though Subclipse and Subversive are a bit weak compared to the gold standard (that would be TortoiseSVN), they are much better than what you get with Komodo or Wing. Now that I am on OS X (no Tortoise) this is a bigger issue for me than it used to be.
• PyDev Extensions has refactoring support now, too.
• Komodo has limited support for completion inside django templates. Which is impressive, since the commands allowed in django templates aren't really Python, which is to say that you can't just use the same completion support that you use for normal Python code.
• Mako template support with completion, anyone?
• The latest versions of Komodo and Wing both integrate unittest support. Wing also supports doctest out of the box. Meaning, you click a button, your tests run, you get a pretty summary with click-to-go-to-the-source-of-the-error support. This might get me to finally upgrade to Wing 3. It's not that "python test.py" is so hard, so much as I do it so often that even a little more convenience adds up.

"Mesh is the only thing that really makes sense out of a Yahoo
acquisition to me. Yahoo has rich content services—and they're
everywhere. If Microsoft could plug Mesh into that infrastructure,
fast, and flip the switch "Wow!" Imagine, for example, Mesh making
Flickr photos instantly available to all your PCs, cell phones and
TVs. Software plus hardware plus services."

But isn't that what the internets are for?

http://www.microsoft-watch.com/content/web_services_browser/yahoo_between_a_rock_and_a_hard_place.html?kc=MWRSS02129TX1K0000535

I’m sure I read this somewhere recently, but my scratchy memory and command of Google can’t bring it back to me.

Is there a Python idiom for accepting either a file name or a file object as a function parameter?

The closest I can get is this;

def my_function(file_name_or_object):
    try:
        open(file_name_or_object)
    except TypeError:
        file = file_name_or_object
    return file

Any improvements on this are more than welcome.

I’m having trouble with dates. This can be summed up in a couple of high level issues;

1. Date support in relational databases is insane, or at the best inconsistent.

As far as I can tell the ANSI SQL-92 standard defines date, time, interval and timestamp data types. Which doesn’t help when SQL Server only implements something called ‘datetime’ - at least I think so, have you tried accessing any sort of manual for a Microsoft product online? Blimey, I thought billg had embraced this web thing years ago. Oracle has the ‘date’ data type (which is actually a time stamp) and MySQL, well they’ve gone and outdone everyone by implementing DATETIME, DATE, TIMESTAMP, TIME, and YEAR.

2. The Python DB-API does not cope with date data type ambiguity well.

When it comes to the date question the Python DB-API states (and I quote) ” … may use mx.DateTime”, which if you ask me isn’t much of a standard. This needs to change so that all DB-API modules return consistent datetime objects, not such a big issue as datetime has been part of the standard library since, what, Python 2.3?

Sadly even if we fix this it won’t work with Sqlite as it doesn’t consistently support data typing. In my experiments regardless of what sort of date you insert into the database you get a unicode string back. Don’t believe me? Try this in Python 2.5;

>>> from sqlite3 import dbapi2
>>> db = dbapi2.connect('test_db')
>>> cursor = db.cursor()
>>> cursor.execute('create table date_test (id integer not null primary key autoincrement, sample_date DATE NOT NULL)'
>>> stmt = "INSERT INTO date_test (sample_date) VALUES (?)"
>>> cursor.execute(stmt, (1234, ))
>>> import datetime
>>> cursor.execute(stmt, (datetime.date(2008, 3, 10), ))
>>> cursor.execute(stmt, ('My name is Earl', ))
>>> db.commit()
>>> cursor.execute("SELECT * FROM date_test")
>>> results = cursor.fetchall()
>>> for item in results:
...     print item[1], type(item[1])
1234 
2008-03-10 
My name is Earl 
>>>

But note that it is fine for integers.

3. The people writing the Python standard library modules are on crack.

Outside of the database world and within the batteries included Python standard library some modules use datetime, others time and there are even uses of calendar.

O.K. I’ll accept that maybe the module authors aren’t on full strength crack, because the time module just exposes underlying posix functions. But the people who wrote those were on something strong and hallucinogenic. I table the following function signatures from section 14.2 of the Python Library Reference 2.5 as an example;

strftime(format[, t ])
strptime(string[, format ])

This has bitten me twice in the last twenty four hours and frankly I’m not happy.

I appreciate that there are historical reasons for having inconsistent function signatures but can someone please fix this in Python 3.0. All we need is a single module that can access the underlying system clock and then convert between a number of different representations of that and other epoch driven dates. How hard can it be? As far as I can tell this is not part of the proposed standard library re-organisation. I think it should be.

• Debugging IronPython Code with Visual Studio
  

Michael Nygard posted about SAP's SOA capabilities. I wasn't sure if he'd appreciated their complexity, but he does emphasize that in a comment on this post here.

We're talking really complicated, WS-*-ish bits. My advice from hands-on experience two years ago would be: stop; walk back slowly, slowly; now turn and run; faster.

Not the kind of good system attributes I think of when reading Michael's book.

As mentioned on the last post I am moving the current repository, an html page, to an actual Git repository on github.com. The link to the repository is

http://github.com/nuin/beginning-python-for-bioinformatics/tree/master

and it can be accessed by anyone. There are only a handful of scripts there but I am slowly adding more comments to the scripts and moving them to github. The web interface at github is pretty nice and the code can be viewed on the website with a nice code highlighting, for example. Also there is an RSS feed to receive updates, commits, etc.

How to create a local copy

Git is very easy to use and it is very simple to create a local copy of the repository on your local machine. Git is available on most systems as a command-line utility (there is a gui but I haven’t used yet) and to have an updated copy of the Beginning Python for Bioinformatics, two commands are needed.

- first you have to clone the repository

$ git clone git://github.com/nuin/beginning-python-for-bioinformatics.git

that will create a beginning-python-for-bioinformatics directory wherever you run git (this only needs to be done once)

and to keep the clone updated

$ git pull

from inside the clone directory.

Any questions please let me know.

In February, Harvard’s Faculty of Arts and Sciences (FAS) unanimously approved an Open Access resolution, committing to make all its research available through a public repository. It was the first US college to do so.

Yesterday, Harvard Law School unanimously voted to become the first US law school with the same commitment.

Over a scant few years, Harvard Law pulled together Larry Lessig and Jonathan Zittrain, and recently recruited both Yochai Benkler and Cass Sunstein. These are, along with folks like John Palfrey, the finest legal thinkers of their generation. I am incredibly hopeful about the kind of cyberlaw activism and trendsetting we’ll see with these minds all sharing an affiliation.

sudo a2enmod ssl

openssl genrsa -out server.key 1024

openssl genrsa -des3 -out server.key 1024

openssl req -new -key server.key -out server.csr

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

sudo cp server.crt /etc/ssl/certs
sudo cp server.key /etc/ssl/private

SSLEngine on
SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key
SSLVerifyClient require
SSLVerifyDepth 1
SSLCACertificateFile /etc/ssl/certs/server.crt

openssl pkcs12 -export -out client_cert.pfx -in server.crt -inkey server.key\
-name 'Certificate Name'

openssl pkcs12 -clcerts -nokeys -in client_cert.pfx -out client_cert.pem
openssl pkcs12 -nocerts -in client_cert.pfx -out client_key.pem

openssl rsa -in client_key.pem -out unsecured_client_key.pem

I’m delighted to release Paver 0.7. If you missed my original announcement, the short story is that Paver is a new build, distribution and deployment scripting tool geared toward Python projects. My original announcement and the new foreword to the docs explain the motivation.

Ben Bangert and others pointed out a giant documentation bug in 0.4: there was a fair bit of reference doc but no doc that said “here’s how you get started with Paver”. Now there is: Paver’s Getting Started Guide.

Paver 0.7 is a big step up from 0.4 (hence the version number bump). I implemented one of the two major features I had planned for 1.0: distutils/setuptools integration. It’s really cool. Have you ever wanted to just slightly change how “sdist” or “upload” or “develop” worked? Now you can, just by writing a function in your pavement.py file. And don’t worry, you don’t need to duplicate anything between setup.py and pavement.py. It all just moves into pavement.py and Paver can even generate a setup.py file for you, since most people are use to the common “python setup.py install” command.

I’ve gone even farther than that with making it easy to use Paver and not annoy users that don’t yet have Paver. Paver can create a small zip file of Paver’s core bits so that “python setup.py install” will work just fine even for users who don’t have Paver installed. Paver can also create a virtualenv bootstrap script for you, so that users don’t necessarily need to install your package on their systems in order to use it.

Paver’s got new documentation tools that work great with Sphinx. It’s now easy to mark sections of sample code files and then include those sections in your documentation, using the built-in version of Ned Batchelder’s Cog.

And I’m definitely eating my own dogfood. Paver is built using Paver itself and the source distribution includes the paver-minilib so that setup.py install should work fine (let me know if it doesn’t!) The new Getting Started Guide uses the new documentation tools.

There are even more changes than these, and you can look at the changelog for the full list. Note that if you’re using Paver 0.4, there are a couple of trivial breaking changes.

ShareThis

• OpenTk - the Open Toolkit

... And it's just the unittest I wrote for an application I'm cooking up:

Python is consuming much in the way of processor time. I <3 the processing module. All the workers are doing is tossing integers from one queue to another (in, out, result) and doing random multiplication on those integers.

Delicious.

Live or semi liveblogging conferences has been getting more and more difficult for me to do. The combination of meetings, networking/parties, and photographs means that it takes longer to assemble the requisite material. Here’s a bit on CommunityOne, which took place on Monday.

Many people (mostly Sun folks) have been asking me if this is my first JavaOne. My answer is, “it’s not, but it is my first one in ten years”. It’s been quite some time since I’ve been to a conference run by a big company like Sun (as opposed to an O’Reilly or open-source community conference). Even though the basics are the same, I definitely feel a kind of culture shock. I was asked to be on a panel during the general session, first thing in the morning, in order to get miked up and to run though the flow. Production values are much higher than I am used to. I keep thinking of CommunityOne as a small event, but in reality it is huge. I am told that registration was around 5000 people, which is twice the size of OSCON, which is the largest conference that I’ve been to in the last 4 or 5 years. Some pictures might help with the scale and production values:

Smashing things makes for cool demos - you can watch the video replay if you like.. I’ve been paying more attention to ZFS ever since Theo Schlossnagle sat with me and a few other people in a bar at ApacheCon in Atlanta last year. We were talking about the voracious storage needs of photographers, and Theo was really singing the praises of ZFS. There were so important things that happened to ZFS for OpenSolaris 00805 (which was launched at CommunityOne). The most important is that you can now boot off of a ZFS volume. I hope (but don’t know for sure) that the work that made this possible will make it possible for Macs to boot off of a ZFS volume. My photo storage is getting all fragmented, and I could really put ZFS to good use. I suppose that I could build a ZFS storage appliance based on OpenStorage, but at the moment that is more work that I want to do.

I spent much of the rest of CommunityOne at the Redmonk unconference. I was drafted for an impromptu discussion on dynamic and other programming languages, which included a drop in from David Pollak, developer of the very cool lift framework for Scala, and organizer of the Scala liftoff which is happening on Saturday, right after JavaOne. There was also a very active session on Twitter - probably the biggest of the unconference. Jim Evans Edwards from Twitter came along to participate in that one

I have a bunch more photos from CommunityOne. At the rate that things are going, I will probably just do a single post on JavaOne. There are plenty of other people doing liveblogging, for those who need a bigger information flow.

Update: corrected Jim Edwards’ name. Thanks to @monkchips

Yesterday I successfully used map() and lambda without having to look @ the documentation (and yet I link to the documentation)!

Something like:

some_url = "http://www.foo.com/"
things = ["foo", "bar", "baz"]
urls = map(lambda x: some_url + x, things)
for u in urls:
    print u

UPDATE: Sorry based on the comment by ‘baoilleach’ I feel compelled to update the code above using map() and lambda with a list comprehension as suggested.

some_url = "http://www.foo.com/"
things = ["foo", "bar", "baz"]
urls = [some_url + t for t in things]
for u in urls:
    print u

And don’t ask why I wasn’t using urlparse, I swear I have a good reason.

Another little thing that I think I was too dense too get was my misconception that lambda could only take one argument. I don’t know where I picked that up could be similar to someone thinking that tuples could only have two items (I’m just kidding Pam) - two-ples, ya know?

Simple lambda influenced by what Mr. I says these days:

>>> mine = lambda n: n.capitalize() + " is mine!"
>>> mine("book")
'Book is mine!'
>>> mine("shirt")
'Shirt is mine!'

But what about more than a single argument. Easy:

>>> huh = lambda what, who: what.capitalize() + ", " + who.capitalize() + "?"
>>> huh("cow", "daddy")
'Cow, Daddy?'
>>> huh("cat", "mommy")
'Cat, Mommy?'
>>>

I don’t use lambda’s very often, most of the time if I’m going to write a lambda I just write a function. Anyway, small victory for me.

I now uploaded the documentation for Jinja2 to the website for those of you who are eager and want to play with it :-) On jinja.pocoo.org you have now the choice to chose between Jinja1 and Jinja2.

The new docs are powered by Sphinx and Jinja2 with a custom templating bridge.

Read the documenation.

I opened a Twitter account for the PyCon Italy conference. I will try to keep updated as soon as things come up and the conference starts on Friday.

http://twitter.com/pyconit

Often I have an iterable i want to group. For example a list of integers and i want to process two at once. That’s a pretty nice idom I found in the documentation translated to itertools:

from itertools import izip, repeat

def batch(iterable, n):
    return izip(*repeat(iter(iterable), n))

Use it like that:

>>> for key, value in batch([1, 2, 3, 4], 2):
...  print key, value
... 
1 2
3 4

After 6 months, PyAWS 0.3.0 is eventually released. You can check out the tar ball here.

I almost abandoned this project as I found the XSLT approach is more appealing: ideal for AJAX application and easy to integrate via simplejson in the server side. Furthermore, I joined Microsoft, moved to Canada, and had less spare time to work on less interested hobby work. The last straw is the unexpected complicity of the the BIG FAT refactory.

Until recently, I got the email from one PyAWS user, he reported a bug on unexpected result of ListLookup operation. It is so good to hear from some users that this library still benefits somebody in the world. So I picked it up, completed the refactory and released it today. The library still in active development, the code style stinks, the document sucks and most of all, testing is lacking — I would explain it for a little bit here.

I am a big fan of TDD personally, and we have respected testing troops to help building our products in MSFT as well. However, the complexity of PyAWS is far beyond my capacity: there are tens of operations and twenties of response groups, and response groups may combine, that make it extremely difficult to cover all the paths. To make it worse, the AWS is dynamic, there is no guarantee that the consecutive queries would return the same result. I may consider automation to facilitate the unit tests. If you have better ideas, please leave a comment here.

Share This

I am transferring the current repository, which is “hosted” in a static page on th blog, to a Git repository on github.com. I am posting soon a quick guide on how to access it. I am also making all relevant entries from the blog into stand alone html pages for off-line browsing.

Anyone with Git experience the link to the repository is here (not many commits yet, but it is coming).

Guido just shot out email to the dev list(s) announcing he has accepted Christian Helmes' Per user site-packages directory PEP. This is awesome as it will provide a lot of functionality for those of us who don't want to sully the system package tree with modules we install. It could also help simplify things like zc.buildout and virtualenv.py

For giggles, you should read the discussion thread - please ignore my less-than-caffenated contribution to it.

I've hardly played Lord of the Rings Online at all the last six months and not at all the last three.

My only copy of Windows is a VMware Fusion instance and LOTRO doesn't work on VMware Fusion. That is...until now.

I was excited to hear that the new VMware Fusion 2.0 beta 1 supported pixel shaders in DirectX 9 and I wondered if that meant LOTRO would work. I downloaded the beta, which JUST WORKED with my existing VM (which wasn't even shut down). I spent an hour or so updating LOTRO but my first attempt to start the game failed.

The error message was different, though. Instead of being about the graphics adapter it was a complaint about a Game Error 127. A Google search revealed this post and so I tried making the config change they suggested there.

And BINGO! I can now run Lord of the Rings Online on VMware Fusion!

I haven't tweaked the settings yet to see if it's playable but I'm hopeful.

VMware, you are amazing!

UPDATE: I played LOTRO for a little bit tonight. I had to turn sound off as it was too jittery. Graphics were fine, though, on Low detail and low resolution. The mouse look problem mentioned in the comments was easily fixed by just going to options and changing the mouse look sensitivity.

Python Meetup in London tonight. See you there!

The excellent 42 Topics blog has a post entitled Popularizing Django — Or Reusable apps considered harmful which makes (or attempts to make) the case for packaged apps over reusable apps.

He raises some good points, although of course the packaged apps he's talking about still use reusable apps so he's not actually talking about there being a problem with reusable apps per se, just that there should be packaged apps as well.

I mentioned the django-hotclub group in a comment on that post as I'd really like the discussion to take place there.

I also, in that comment, mention something I'm working on tentatively called Tabula Rasa. (I'm toying with a Greek name rather than Latin but something tells me people are more comfortable with tabula rather than grammateion)

Basically, the goal of Tabula Rasa is an out-of-the-box Django-based website with everything but the domain-specific functionality.

So far it's just my new django-email-confirmation app tied together with password change and reset, login/logout, with the beginnings of a tab-style UI. There's a ton more I want to refactor out of my existing websites to put into it as well as adding support for OpenID and the stuff I'm starting to do for django-friends.

Even if one doesn't use Tabula Rasa as the starting point of a website, I'm hoping it will prove very useful for another goal, namely a "host" project to develop and tryout reusable apps.

One of the challenges I know I've always had with writing or trying out reusable apps is the need for a project to provide the scaffolding.

So Tabula Rasa will hopefully serve that dual purpose.

The initial code is available at http://code.google.com/p/django-hotclub/ under /trunk/projects/tabularasa

I hope to have a running instance online soon.

There are many times when the programming task at hand is to iterate over some semi-structured text, transform parts of that text in some way, and reintegrate those transformed parts back into the original text.

Typically using a regular expression with re.sub and a callback function, but sometimes you want a bit more control of the process (especially over those parts that dont match the regex). Usually my solution is to write a one-off function that does it, but today I had to write that function yet again and decided to generalize it and post it here.

To be completely honest, this post is more for my own archival purposes than for the internet as a whole, but if anyone else finds it useful, then I'm ecstatic.

def re_parts(regex_list, text):
    """
    An iterator that returns the entire text, but split by which regex it
    matched, or none at all.  If it did, the first value of the returned tuple
    is the index into the regex list, otherwise -1.

    >>> first_re = re.compile('asdf')
    >>> second_re = re.compile('an')
    >>> list(re_parts([first_re, second_re], 'This is an asdf test.'))
    [(-1, 'This is '), (1, 'an'), (-1, ' '), (0, 'asdf'), (-1, ' test.')]

    >>> list(re_parts([first_re, second_re], 'asdfasdfasdf'))
    [(0, 'asdf'), (0, 'asdf'), (0, 'asdf')]

    >>> list(re_parts([], 'This is an asdf test.'))
    [(-1, 'This is an asdf test.')]

    >>> third_re = re.compile('sdf')
    >>> list(re_parts([first_re, second_re, third_re], 'This is an asdf test.'))
    [(-1, 'This is '), (1, 'an'), (-1, ' '), (0, 'asdf'), (-1, ' test.')]
    """
    def match_compare(x, y):
        return x.start() - y.start()
    prev_end = 0
    iters = [r.finditer(text) for r in regex_list]
    matches = []
    while iters:
        if matches:
            match = matches.pop(0)
            (start, end) = match.span()
            if start > prev_end:
                yield (-1, text[prev_end:start])
                yield (regex_list.index(match.re), text[start:end])
            elif start == prev_end:
                yield (regex_list.index(match.re), text[start:end])
            prev_end = end
        else:
            matches = []
            for iterator in iters:
                try:
                    matches.append(iterator.next())
                except StopIteration:
                    iters.remove(iterator)
            matches = sorted(matches, match_compare)
    last_bit = text[prev_end:]
    if len(last_bit) > 0:
        yield (-1, last_bit)

1. Namespaces. In the world of most Javascript, everything is global or function-scope, there seems to be no use of namespace or module-style development at all. It's possible to simulate namespaces by wrapping entire modules in an object definition, but this is usually too much of a hassle for practicality.
2. Objects. Javascript's object system is, well, crap. Sure, it's ad-hoc like Python, but it has no formal way of creating simple constructors. You can use the "this" object in a function, or create a new instance of "Object" to append methods to, but there are no standards for constructing an object.
   
3. Objects do not support destructors. Many might argue that a memory-safe language like Js doesn't need them, but destructors have more use than freeing memory. They serve a very useful purpose in a lot of OO designs, memory-safe or not.
4. Javascript does not support associative arrays! You can use the "Array" object, or even the "Object" object to assign associative values to using the index operators, but this is deceiving... Any time you use "obj['index'] = something" you could replace it with "obj.index = something" which causes major problems with naming collisions. This style of object/hashtable is bad, there should be a clear distinction between the two.
   
5. Javascript is not cross-browser compatible... Don't get me started...