Quickly on Ubuntu: Making it Quicker with PyDev

I was pleased to see that the team at Canonical have been making some progress with their application development documentation and the Unity API.

There's clearly strong emphasis on helping beginners to programming get their hands dirty and hack some code. I think it's great to see - this is what *nix is all about.

Looking over things, the toolstack they're recommending contained no major surprises. Python is a popular scripting language with a solid range of libraries, which is nice and easy for beginners to pick up. And Canonical's made a tool called Quickly to help budding devs hit the ground running, by creating some of the tedious boilerplate around persisting preferences, creating key GUI windows, logging, etc. Quickly also handles some of the packaging, and can upload to a PPA.

I thought it would be fun to give the stack a test run.

But, alas, I had somehow missed one of the items on that toolstack page. I must have seen it, I suppose... I must have mentally filtered it out. Because it didn't seem possible. But when you type quickly edit to bring up the code, there it is, in the digital flesh.

The IDE Canonical is pushing, is... Gedit.

They can't be serious

But they are. The biggest argument I have with Canonical over their default toolstack is unquestionably the decision to set gedit as the editor. It's just a basic text editor. For any primitive GUI-based text editing needs I have, I generally use Leafpad or Gvim, which can get the job done quicker.

Gedit's biggest features are - get ready for them - tabs, syntax highlighting, find & replace (just about...), and plugins (of which there are some of note, but alas, too few). This may be enough to give Microsoft's good ol' Notepad application a good run for its money, but it falls short of what is expected in the 21st century for application development.

Come on Canonical. You must be jesting, right?

An Ode to Power
 
I'd personally be vastly more productive in vim than gedit, as would many experienced users. But the user base targeted by Quickly is unlikely to want to delve too deeply into NERD tree, especially after grappling with one of the first textual mode-based interfaces they'll have seen in a while! Let's face it; on the Windows side, Notepad++ is so vastly superior to gedit that it almost doesn't bear comparison, and yet few Windows devs would deign to use that as an IDE.

To make matters worse, quickly edit starts off by opening all the files in your project dir simultaneously. It doesn't take much coding before this becomes totally unwieldy.

I stuck it out for a while, but I had to find something better. After trying numerous Python IDEs in the repos and finding each to come up short in one way or another, I came across PyDev - Python support for Eclipse.

Preparing PyDev

+ Install Eclipse (I tend to grab the latest from Eclipse directly, since the repo copy is typically out-of-date, and it comes with its own updating functionality anyway).
+ Add PyDev's update site http://pydev.org/updates to Eclipse (Help -> Install New Software -> Add)
+ Install PyDev

A Few Prerequisites

It took me a good hour to get my project working properly with PyDev, so let me try to save you some of the hassle. 

1. Create a new project (PyDev -> PyDev Project)
2. Provide the correct project name (with the same spelling and capitalization as your source dir. If you change anything about it, you may run into difficulty having it locate your source files)
3. Configure your interpreter (autoconfigure should work ok)
4. Allow it to create your project. If your source files don't appear, go back to step 2 and check that project name and path carefully.
5. Go to project properties-> PyDev - PYTHONPATH -> External Libraries, and add the <projectname>_lib source folder
6. Window -> Preferences -> PyDev -> Interpreter - Python
  a) Under Libraries, add the <projectname>_lib folder
  b) Under Libraries, add the <projectname> folder
  c) Under Libraries, add the <projectname>'s parent folder

So, e.g. if your project is called foobar, you might have the following: /path/to/foobar/foorbar_lib, /path/to/foobar/foobar, /path/to/foobar. Without this seemingly excessive arrangement, PyDev won't be able to locate the nested imports under <projectname>_lib.__init__.py, and you'll see unreferenced import errors for the <projectname>_lib imports.

7.  Go to the next tab - Forced Builtins.
  a) Add "gi". This is required for PyDev to resolve the dynamically created references to the various GTK3 imports under gi.repository.XXX, such as Gtk, Gdk, GObject and Gio. If you look at /usr/lib/python2.7/dist-packages/gi/repository/__init__.py, you'll see what I'm talking about.
  b) Add "<projectname>". This is required for PyDev to resolve imports internal to your project. Otherwise, you're likely to see "from <projectname> import <projectname>Window" give an Unresolved import error.
  c) Click Apply for PyDev to resolve those gi.repository imports.

You should be set up.

A Stack of Improvements, Fit for a Toolstack

Now you have:

+ inline errors and warnings
+ auto-build
+ auto-indentation 
+ true auto-complete (not just random-word-from-the-same-file suggestion)
+ code formatting & import cleanup
+ refactoring capabilities
+ clickthrough to source
+ testing framework
+ a python console
+ a file browser

and all the other niceties of Eclipse. Isn't that better?

Oh, yes... amidst all that, I forgot to mention. It also has tabs, syntax highlighting, find/replace and plugins. What a relief!

Now the toolstack consists of:

+ Quickly - for boilerplate code to help get new projects started quickly, along with easy integration with Ubuntu's PPAs
+ Glade - for the UI design
+ Eclipse with PyDev - for coding

This is a much more acceptable configuration in my eyes.

I know Canonical is trying to keep it simple for those new to development, but I'm unconvinced that gedit alone is the solution. It can be quite hard to write clean code quickly without at least some of the functionality of an IDE.

To give an example: when I imported the code I'd written in gedit into Eclipse, up popped a raft of warnings for unnecessary import declarations scattered throughout my classes. This might just be an example of my sheer incompetence, but I suspect I'm not the only one who refactors some classes, and omits to remove all the affected imports.

I think, at the very least, the Quickly workflow needs a more featureful default editor, and some mention of the procedure to migrate to an IDE, such as Eclipse with PyDev, or even Vim.

But, credit where credit's due... it does at least support easily switching to Vim, given that quickly edit respects the EDITOR and QUICKLY_EDITOR environment variables. That is one saving grace at least.

</rant><p>Happy hacking!</p>

Troubleshooting - Run/Debug doesn't work in PyDev

Initially, you'll probably only see the following in the Eclipse Console:

pydev debugger: starting
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject

This is easy to fix. Quickly invokes the main method explicitly from bin/<projectname> with:

import <projectname>
<projectname>.main()

Similarly, PyDev needs an invocation of main() - otherwise it's not going to do very much. Open __init__.py in your project and add this at the very end:

if name == "__main__":
    main()

Now when you run/debug it, it should show signs of life before dying with the schema error (see below for background on it). This too is easy to fix - open the run or debug configuration window, and switch to the Environment tab. Add an XDG_DATA_DIRS variable with the value:

/path/to/<projectname>/data:/usr/share

For example, if the root of your project is at /home/bob/src/myproj, the value would be:

/home/bob/src/myproj/data:/usr/share

Troubleshooting - Settings schema not installed

You see the following error when running quickly debug, or python bin/<projectname>, etc:

GLib-GIO-ERROR **: Settings schema XXX is not installed.

I posted a solution to this on AU; in short, set XDG_DATA_DIRS=$XDG_DATA_DIRS:data, assuming you're in your project's root dir. For example:

cd myproj
XDG_DATA_DIRS=$XDG_DATA_DIRS:data python bin/<projectname>

Note that you have to have used quickly run to generate the compiled pyc files, and the bin/<projectname> script, before you can do this!

Troubleshooting - Glade unable to configure label text for GtkComboButton

There are a couple such bugs in Glade, which I believe to be fixed already. The corresponding version may already be in the proposed repos, although I can't remember where I saw this.

These are easy to work around by opening Glade's XML definition at data/ui/<projectname>Window.ui and removing/modifying the extraneous "checkbutton" text. Save your Glade work first, then open the XML file and make the necessary changes, and finally restart Glade.

Troubleshooting - Python memory leaks

I have a cairo DrawingArea which I'm using to display an updating line graph. When I left my app running overnight, I found that Rss (resident pages) had shot through the roof. A great tool to track down memory leaks in Python is heapy. It's not the most intuitive tool around, but it's not awfully difficult to use either.

For a start, edit __init__.py in your project, and:

+ add import signal
+ add signal.signal(signal.SIGQUIT, start_pdb) to the main method, before the call to Gtk.main()
+ add the method declaration:

def start_pdb(signal, trace):
    import pdb
    pdb.set_trace()

Start your application directly with python instead of quickly, as described earlier in this ridiculously long post. Then, from another terminal, send it the QUIT signal to drop to a debugging prompt

kill -QUIT $(ps aux | grep python | grep <projectname> | awk '{ print $2 }')

From there, you can proceed to analyse the Python heap with heapy.

It was immediately apparent from heapy that my problem didn't lie in the Python heap. The Python heap was occupying just 18MB out of a total residency of over 1GB.

Remember that Python sits atop C. So we have actually got 2 heaps here: the C heap, and the Python heap. 

To inspect the C heap, I fired up the old favourite valgrind. Whilst it didn't provide conclusive proof, it seemed to indicate that whilst the GTK3 libs weren't 100% watertight, the big problem was actually with pangocairo - the font library I'm using to draw nice, smooth, legible text.

After I dramatically reduced the frequency of invoking key pangocairo methods, memory utilisation stabilised over prolonged periods. If I get time, I'll try to confirm the problem is indeed in pangocairo, and track where it's leaking. (Note that pangocairo is a very useful library, and I don't want to discourage you from using it by any means - just keep one eye on the system resources!)

Pink 650D RAW images under Ubuntu

Underwater photos are usually blue

The versions of libraw and dcraw currently in the Ubuntu repos have a slight problem. When accessing RAW files from Canon's latest 650D, they display them entirely in pink - which probably isn't what you want.

The problem is simple enough: the versions of dcraw/libraw used by all the RAW processing apps in Ubuntu 12.10 (aside from eog - so Shotwell, UFRaw, RawTherapee, etc.) don't have the 650D listed yet, so the ICC profiles aren't getting properly interpreted.

One solution is to rebuild your applications, such as Shotwell, with the appropriate versions of the underlying raw libraries. That can be a pain, so I took the easier option, and opted to convert the images to JPEG first, thereby circumventing the issue.

sudo apt-get install libjasper-dev libjpeg-dev liblcms1-dev graphicsmagick
wget http://www.cybercom.net/~dcoffin/dcraw/dcraw.c
gcc -o dcraw -O4 dcraw.c -lm -ljasper -ljpeg -llcms
sudo mv dcraw /usr/local/bin/dcraw
echo alias rawtojpg="\"find . -name '*.CR2' -print0 | xargs -0 -n 1 -P 8 -iIMG gm convert IMG -format jpg IMG.jpg\"" >> ~/.bashrc
source ~/.bashrc

At least it's better than pink

Thereafter, you should be able to use the primitive alias rawtojpg to convert all .CR2 files in the current directory to JPEGs (leaving the original CR2 files as-is). Note the -P 8 flag, which runs 8 conversion threads in parallel - adjust as appropriate. You can obviously change the alias to a proper function which takes some args too, but the above was sufficient for my purposes.



Update, 8th August 2013

See my comment below for a complete solution to this problem.

Assuming you've already followed the preceding instructions in this post, all you need to do is install gimp-dcraw; then you'll be able to open your 650D CR2s with GIMP:

sudo apt-get install gimp-dcraw

Has the Apple bubble burst?

This week, the Apple share price tanked. In one day alone, it lost market capitalization of some 35bn USD.

Apple Inc. hit its all-time peak in September after stunning growth in the 1st quarter of 2012. Since then, it's just been sliding back downhill (barring a brief reversal in the last half of November, which has just been wiped off the board this week).

I'll leave the detailed analysis for others, but there are a few drivers from the tech side that seem worthy of comment.

• The top of the hill: With such a large percentage of mobile phone and tablet sales, and with an ever increasing number & quality of players in the market, it's hard to see how they could make significantly greater inroads here. It's now more about maintaining customers than onboarding new ones. What Apple needed was a new market; a new product, and I don't see one on their horizon at present.
• Litigation: Apple has been suing companies left, right and centre. These cases haven't been to uphold moral principles, or to protect complex designs from illegal duplication; they've simply been attempts at money-grabbing (from the largest US corporate by capitalization). Now the victims have started to fight back against their attacker, turning Apple's insatiable greed into unappetizing risk. Meanwhile, end users stuck in the middle are increasingly becoming disillusioned with Apple's aggressive tactics, and seeking alternative companies to support.
• Google Maps: If you're going to sue them, you'd better not be reliant on their services. By ditching Google Maps and running with their homegrown solution, Apple removed one risk from its risk register, but in doing so created a certainty: an immature product. No matter how much Apple may improve Siri, or how much polish they put on the icon set, it isn't going to make up for taking away from users what they already had.
• Gradual improvement: This sounds good; it's what we often see with each iteration of the Linux kernel: a number of small improvements, which may or may not impact users much, depending on their individual interests. The problem in terms of commercializing it is that few people need or want to upgrade for each and every small set of improvements. Yet, since Apple has reached near-saturation in the western markets, it is reliant on users upgrading frequently. It either has to persuade users that they really do need those extra pixels on the screen, or it has to continually innovate. This is where they've slowed down, and understandably so. How much more can and should be packed into devices in everybody's pockets?
• Competition: Apple led the way with the all-round quality of their first music players, smartphones and tablets. Competition existed, but it was either inferior or small-scale. Now, that simply isn't the case: competitors have caught up, and in some instances actually surpassed Apple. For example, when asked, I presently recommend the S3 for smartphones, and the Galaxy Tab for tablets. Again, it comes back to a lack of innovation on Apple's behalf. The phrase "sitting on one's laurels" comes to mind.
• Remember from whence you came: Before Apple's runaway success with the iPod and its succeeding foray into small form factor mobile devices, it was best known as a niche hardware/software provider. Its primary market was for digital design artists across audio and visual spaces. This was back in 2001. Roll forward eleven years to 2012, and the PC space has remained largely similar; Apple hasn't made any major inroads into mainstream computing (it has gained market share, as some buyers of Apple mobile devices buy into the Apple ecosystem, but the overall impact to the global PC market share is small - we're talking a few percentage points change). Without the success of the mobile device space, Apple is just a comparatively small, niche hardware/software provider. It is reliant on the mobile space's - and iOS's - continued growth. If, for example, desktop OSs were to become popular on mobile (Ubuntu with Unity springs to mind), would there still be room for iOS?

Does Apple still have some cards hidden up its sleeve, or is it floundering precariously on the proverbial ledge, mourning the loss of direction from its most inspirational and charismatic late leader?