Microthoughts

New CLI approach for Python applications

Fri, 25 Sep 2009 17:09:31 +0100

I’m not sure whether I’ve just reinvented the wheel, but … a new way of building command-line interface in Python!

Well, it probably won’t work for everything. Where it works at it’s best is when you have a number of modules, which meant to do something, and you don’t want to build the CLI interface to each of them. Say, in my case — I have 5 modules for the running various machine-learning tasks, which act as wrappers around various Java applications (written by me and third-party). Each module has a number of functions, which have something in common — they take input file(s), process it and write output to another file.

Now, let’s get to the code.

tagger.py - performs feature extraction and invokation of Mallet tagger

Actual code is not included — just methods’ names.

import os
import sysfrom common import *
def extractFeaturesForTagger(filename, output, jrunner = None):
...def invokeTagger(featuresFile, modelFile, train = True, output = None, jrunner = None):
...
def despatch(args):
if len(args) < 3:
return False    cmd = args[2]

if cmd == 'fe':
try:
(ifile, ofile) = args[3:]
except:
return False        extractFeaturesForTagger(ifile, ofile)
elif cmd == 'train':
try:
(ifile, ofile) = args[3:]
except:
return False
        invokeTagger(ifile, ofile)
elif cmd == 'tag':
try:
(fts, model, ofile) = args[3:]
except:
return False        invokeTagger(fts, model, False, ofile)
else:
print("Couldn't recognise the options: ")
print(args)
return False
    return True
def describe():
return ("tagger", [
("fe", "[I]uafFile, [O]featureVectorsFile", "Extract features for tagger"),
("train", "[I]featureVectorsFile, [O]modelFile", "Train the model"),
("tag", "[I]featureVectorsFile, [I]modelFile, [O]taggerOutputFile", "Tag locations in file")
])

What matters is two functions - despatch and describe. First one actually takes standard sys.argv, assumes that it applies to current module and tries to apply the arguments to it’s functions. When it works, it invokes an appropriate method. When it’s not, it returns False.

Second method returns a tuple of string and list, which, in it’s turn consists of tuples of three elements — subcommand name, parameters and short description. Now when we have this module, let’s get to the despatcher.py:

#!/usr/bin/env python
# encoding: utf-8
"""
despatcher.pyCreated by Roman Kirillov on 2009-09-25.
Copyright (c) 2009 Yahoo! Inc. All rights reserved.
"""
import sys
import osimport tagger
import disambiguator
modules = [tagger,disambiguator]def printHelp():
    helpString = '\nUsage options: %s   \n' % (sys.argv[0])
for m in modules:
(cmd, subcmds) = m.describe()
        helpString = helpString + '\n' + cmd + '\n'        for scd in subcmds:
(s, params, descr) = scd
            helpString = helpString + (s + "\t").rjust(15) #  + params + " -- " + descr + "\n"
            helpString = helpString  + params + " -- " + descr + "\n"
    print(helpString)def canIHaz(val):
if not val:
        printHelp()
        sys.exit(3)
def despatch():
if len(sys.argv) < 2 or sys.argv[1] == 'help':
        printHelp()
        sys.exit(0)
else:
for m in modules:
(cmd, subcmds) = m.describe()
if cmd == sys.argv[1]:
                canIHaz(m.despatch(sys.argv))
if __name__ == '__main__':
    despatch()

While it’s pretty easy to anyone who can read Python to understand, what’s this all about, let’s just show what happens when you run despatcher.py without parameters:

[kirillov@chiark integration]$ python despatcher.pyUsage options: despatcher.py   
tagger
            fe  [I]uafFile, [O]featureVectorsFile -- Extract features for tagger
         train  [I]featureVectorsFile, [O]modelFile -- Train the model
           tag  [I]featureVectorsFile, [I]modelFile, [O]taggerOutputFile -- Tag locations in filedisambiguator
            fe  [I]uafFile, [O]outputFileName, [I]useTaggedLocations (true/false) -- Invoke disambiguation feature extractor
        import  [I]featureVector, [O]malletImport -- Import feature vector to Mallet format
         train  [I]featureVector, [O]modelName -- Train disambiguators model
      disambig  [I]featureVector, [I]modelName, [O]output -- Run trained disambiguator over the feature vector
     normalise  [I]featureVector, [O]normalisedFeatureVector -- Normalises the feature vector

Pretty easy, huh? Of course, still needs a lot of tuning and polishing, but it feels so much better to have a centralised dispatcher, which takes care of printing help and handling options — rather than having a one for each module?

Posted via web from Roman’s blog | Comment »

VoIP calls via 3G from your iPhone

Tue, 22 Sep 2009 08:15:55 +0100

What if you’re away from any wifi or desk phone with a fancy “All World Unlimited” price plan (i.e. your work phone), and you still need to make that long international call urgently? Apple says there’s nothing you can do about it and you must use your provider’s international tarif. Independent developers who don’t bother putting their app to AppStore disagree. 3G is just another source of the Internet, and if Steve, O2, AT&T & co doesn’t want youto use it the way you like - worse for them. Because tiny app Siphone (apparently, SIP + telephone) works nicely via your 3G connection, and all you need is a SIP account with some money on it (I use voipcheap.com for that backup purpose - they’re crap, but none of them is any better). This morning I had a call with my wife (who’s meeting with some of her clients in Moscow) and chatted for 15 minutes; guess how much should I pay if I use O2’s international tarif?

Anyway, Siphone is a little gem and if you have your iPhone jailbroken, you have absolutely no reasons not to install it. Do it. Now.

Roman Kirillov
http://sigizmund.com

Sent from my iPhone

Posted via email from Roman’s blog | Comment »

New Yahoo! marketing campaign: it's you!

Mon, 21 Sep 2009 18:34:05 +0100

Totally completely utterly amazingly breathtakingly awesome! Proud to work for Y!

Posted via email from Roman’s blog | Comment »

Two questions to Posterous ((tag: posterous)

Mon, 21 Sep 2009 10:54:13 +0100

Hi guys, First - thanks again for a nice service, themes are really cool and I’m really going to do some heavy-weight customisation over the weekend! Now, two questions:

1. Is there any way to hide past of the post behind the “read more” link? Like in LiveJournal or ” in WordPress? That may definitely help to make Posterous more solid and professional blogging tool. 2. Is there any way to convert tags into twitter hashtags? So, if I tag this post (you see - I’m CCing post@posterous.com!) with “posterous” tag, can it be converted into #posterous on Twitter?

3. (oi! there’re three questions actually!) – can we have an option of sending content which will be processed by Markdown OR at least sending HTML source? Say, I want to include snippets of code into my post, so I’d prefer to have `int a = 128` or at least int a = 128; feature. Thanks again!

Cheers,
Roman

Posted via email from Roman’s blog | Comment »

Edit in TextMate functionality in Firefox

Sun, 20 Sep 2009 13:25:44 +0100

Do you like “Edit in TextMate” command you can use in any Cocoa edit window, and write the text in your favourite text editor? (of course, TextMate that is). I like it too, and since I’ve switched to Firefox (after 1.5 years struggling with Safari and getting finally tired of it being SO slow sometimes!) I missed it a lot. Well, now you can do it, too – just follow the advice here: http://blog.circlesixdesign.com/2007/02/23/textmate-firefox/ Little tip: there’s no need to create any symbolic link, you can simply specify `/usr/local/bin/mate` as your editor, or, if you didn’t install `mate` command when installing TextMate (which I highly recommend to do), you can use `/Applications/TextMate.app/Content/MacOS/TextMate` to achieve nearly identical results.

Posted via email from Roman’s blog | Comment »

Mesmerising Flash video/animation/game

Sun, 20 Sep 2009 09:37:53 +0100

Mesmerising Flash Application from Roman Kirillov on Vimeo.

Try it yourself, really.

Posted via web from Roman’s posterous | Comment »

Debugging Hadoop applications using your Eclipse

Thu, 17 Sep 2009 16:30:47 +0100

Well, it can be annoying - it can be awfully annoying, in fact, to debug Hadoop applications. But sometimes you need it, because logging doesn’t show anything, and you’ve tried anything but still cannot get under the Hadoop’s cover. In this case, do few simple steps.

1. Download and unpack Hadoop to your local machine.

2. Prepare small set of data you’re planning to run the test on

3. Check that you actually can run Hadoop locally, something like this (don’t forget to set $HADOOP_CLASSPATH first!):

bin/hdebug jar yourprogram.jar com.company.project.HadoopApp tiny.txt ./out

4. Go to Hadoop’s directory, and copy file bin/hadoop to bin/hdebug

5. Now, we need to make Hadoop start in debug mode. What you should do is to add one line of text into the starting script:

Yes, here’s it. Copy it from here:

-Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y

What does it say basically is an instruction to Java to start in debug mode, and wait for socket connection of the remote debugger on port 8001; execution should be suspended after the start until debugger is connected.

Now, go and start your grid application like you did in step 3, but now use bin/hdebug script we’ve created. If you’ve done everything correctly, program should output something like this:

Listening for transport dt_socket at address: 8001

and wait for debugger. So, let’s get it some debugger then! Fire up your Eclipse with your project (likely you have it opened already since you’re trying to debug something) and add new Debug configuration:

After you’ve set everything up, click “Apply” and close the window for now – probably, you’d want to set some breakpoints before starting the actual debugging. Go and do it, and then simply choose created debug configuration - and off you go! If everything worked properly, you should soon get a standard debugger window, with all the nice things Java can offer you. Hope it’ll help some of us in our difficult business of writing distributed grid-enabled applications! :)

Posted via email from Roman’s posterous | Comment »

My O2 iPhone App - easy and convenient, does what it says

Tue, 15 Sep 2009 14:44:06 +0100

So, you would really like to now, how much longer can you use your inclusive minutes and text, or want to see, when your next bill is due or how much did you pay the last time? There’s an app for that! Recently released My O2 app is a tiny brilliant - does what it says, easily and simply – no more, and no less.

Main screen simply shows you the most important information – well, for me it’s not that bad, as the last bill was just yesterday and I am unlikely to put the needle to the left-most position. But, I can easily see what I want to, and that’s pretty much enough. You can also view your last bill summary, your recent charges (I found out how much one minute costs me when I’m in Spain!) – and even view your tariff details, which is something totally amazing if you need just a quick reference, on what you’re paying. Honestly, O2 - you did a great job and my congratulations are well-deserved!

See and download the full gallery on posterous

Grab it while it’s hot! (via O2 Blog)

Posted via email from Roman’s posterous | Comment »

My O2 iPhone App - easy and convenient, does what it says ((tags: geek, iphone, apple)

Tue, 15 Sep 2009 14:41:35 +0100

Grab it while it’s hot! (via O2 Blog)

Posted via email from Roman’s posterous | Comment »

Me and Natalie on our friends' wedding

Tue, 15 Sep 2009 10:38:46 +0100

I bet you’re surprised I do not look as smart as the rest of the guests (and not even nearly as smart as my beautiful wife). Well, that’s because I’ve been the most miserable person there – namely photographer, and spend the whole day jumping around with the camera (with two cameras, in fact), so at 12AM when guests were enjoying the Spanish wine and truly delicious food from the hotel’s restaurant, I fought myself to not fall asleep right where I was sitting.

Posted via email from Roman’s posterous | Comment »

A problem for real engineers

Mon, 14 Sep 2009 11:37:06 +0100

On the first of september 100 immortal prisoners with a life sentence are suggested an early release if they can find a solution to a problem. There’s a room with a hanging lightbulb. Every day, starting on the 1st of September, warder will let one prisoner into this room. When prisoner is in, he can see, whether lamp is on or off, or switch it. Each prisoner will be asked a question: “Whether every prisoner has been here?”. If he says “no”, game goes one. If he says yes and this is a correct answer, everyone is released. If he says yes and he’s wrong - everyone is executed.

Warders can choose prisoners randomly, one prisoner can be choose as many times as warder wants. Prisoners do not see each other, cannot talk and cannot see the lightbulb; they can talk only once - on the 1st of September, before the game starts. Find an optimal strategy, estimate, how long it may take for prisoners to be released.

Posted via email from Roman’s posterous | Comment »

Producing Star War effects with Safari 4

Tue, 08 Sep 2009 15:17:31 +0100

Yes, it’s all for real. Look at the video.

Download now or watch on posterous

StartWarsSafari - Computer.m4v (7303 KB)

In my Twitter (http://twitter.com/sigizmund/status/3840531748) I’ve added a link to video-manual, how to produce this breathtaking, but utterly useless effect.

R.

Posted via email from Roman’s posterous | Comment »

I want my iPhone - re-think of Sinatra's hit

Mon, 07 Sep 2009 12:01:17 +0100

Really great to my opinion! Well, maybe there’s just a very little bit of pre-justice.

Posted via email from Roman’s posterous | Comment »

The Boat That Really Rocked

Sat, 05 Sep 2009 22:14:19 +0100

Richard Curtis is absolutely, unbelievably, unbearably and intolerably fantastic. It’s the best movie I’ve seen in the last few years, that’s so bad I didn’t have a chance to see it before. Maybe it’s a first time I’m feeling slightly sorrow I’ve never been here in Britain back then in 60s to see it all alive. That really rocked.

Posted via email from Roman’s posterous | Comment »

New posterous address

Sat, 05 Sep 2009 11:49:47 +0100

OK, I’ve made up my mind - an idea of photoblog didn’t work, so I’ve decided to switch my main domain - http://sigizmund.com - to my posterous account; it is now live and if you’re seeing this post, your address bar probably says “sigizmund.com/something”.

Posted via email from Roman’s posterous | Comment »

WOEID Info

Fri, 04 Sep 2009 19:29:37 +0100

Something what I wanted to do for such a long time - to make publicly available service, which can answer one simple question – what does this WOEID mean? So, I’ve made it - in a way that anyone can put a link to it with a WOEID in URL. Finally, everyone who wants to find out how does that look like, shouldn’t bother getting Yahoo! AppID, read API docs and so on and so forth (I am not saying that reading docs is a bad idea – it’s actually very good and you can save a hell lot of time and nerves by doing that, but sometimes, well … we just don’t feel like it). Anyway, take a look and maybe you will find it useful. In my plans to add a geocoder to the app (so you can type not the WOEID but the human-readable name, such as “London, UK”).

Ah. Almost forgot. You can find WOEID Info right here.

Posted via email from Roman’s posterous | Comment »

Stop being greedy, BOSE - part 2

Wed, 02 Sep 2009 21:34:49 +0100

Griffin’s SmartTalk works at least twice better than Bose’s £30 mobile headset cable - just because it has a button and now I enjoy same level of comfort as when using Apple’s headset - but with incomparably higher sound quality. And I paid only £13.95 shipping included for this - if it’s not a bargain, I don’t know what it is.

Posted via email from Roman’s posterous | Comment »

Windows 7 impressions

Wed, 02 Sep 2009 16:13:23 +0100

OK, so I’ve installed it (in form of a free 90-day preview version distributed by Redmond guys) and it’s running in VirtualBox on my MacBook. It’s running OK, really. Certain things do really make me happy - the taskbar, which uses the screen space in more optimal way is one of them; really nice and streamlined GUI is another. Snipping Tool, a nice little utility for taking screenshots - is just what many Windows users been asking for many years, is not built-in and shipped as a part of operating system - this could be a real win, but … you see - I can make a screenshot of a part of the screen. I can draw something using pen or highlighter. What else can I need? Probably, draw an arrow and write some text - and this cannot be done. So, it’s not win, really – and it lacks only a tiny feature. To be perfectly honest with you, it’s very unlikely that I will use it IRL any time soon – I’m poisoned, poisoned long ago by Apple venom, and there’s no known vaccine for it, but I must appreciate, what Microsoft is doing - 7 seems to work as fast as XP in virtual machine, and it is way faster than Vista - looks like a right thing to use.

See and download the full gallery on posterous

Posted via email from Roman’s posterous | Comment »

My Snow Leopard is finally here

Wed, 02 Sep 2009 10:50:29 +0100

See and download the full gallery on posterous

Posted via email from Roman’s posterous | Comment »

PlaceMaker + Flickr + YQL = PhotoMaker

Tue, 01 Sep 2009 16:46:08 +0100

Today I’ve created and published a PhotoMaker – mash-up, which geoparses articles, using Yahoo! Placemaker and finds photographs on Flickr which are relevant to found places. It uses YQL to glue them together - and initially was created as a demo of these three brilliant pieces of technology. Please take a look yourself at sigizmund.info/PhotoMaker.

Posted via email from Roman’s posterous | Comment »