Quickly on Ubuntu: Making it Quicker with PyDev

I was pleased to see that the team at Canonical have been making some progress with their application development documentation and the Unity API.

There's clearly strong emphasis on helping beginners to programming get their hands dirty and hack some code. I think it's great to see - this is what *nix is all about.

Looking over things, the toolstack they're recommending contained no major surprises. Python is a popular scripting language with a solid range of libraries, which is nice and easy for beginners to pick up. And Canonical's made a tool called Quickly to help budding devs hit the ground running, by creating some of the tedious boilerplate around persisting preferences, creating key GUI windows, logging, etc. Quickly also handles some of the packaging, and can upload to a PPA.

I thought it would be fun to give the stack a test run.

But, alas, I had somehow missed one of the items on that toolstack page. I must have seen it, I suppose... I must have mentally filtered it out. Because it didn't seem possible. But when you type quickly edit to bring up the code, there it is, in the digital flesh.

The IDE Canonical is pushing, is... Gedit.

They can't be serious

But they are. The biggest argument I have with Canonical over their default toolstack is unquestionably the decision to set gedit as the editor. It's just a basic text editor. For any primitive GUI-based text editing needs I have, I generally use Leafpad or Gvim, which can get the job done quicker.

Gedit's biggest features are - get ready for them - tabs, syntax highlighting, find & replace (just about...), and plugins (of which there are some of note, but alas, too few). This may be enough to give Microsoft's good ol' Notepad application a good run for its money, but it falls short of what is expected in the 21st century for application development.

Come on Canonical. You must be jesting, right?

An Ode to Power
 
I'd personally be vastly more productive in vim than gedit, as would many experienced users. But the user base targeted by Quickly is unlikely to want to delve too deeply into NERD tree, especially after grappling with one of the first textual mode-based interfaces they'll have seen in a while! Let's face it; on the Windows side, Notepad++ is so vastly superior to gedit that it almost doesn't bear comparison, and yet few Windows devs would deign to use that as an IDE.

To make matters worse, quickly edit starts off by opening all the files in your project dir simultaneously. It doesn't take much coding before this becomes totally unwieldy.

I stuck it out for a while, but I had to find something better. After trying numerous Python IDEs in the repos and finding each to come up short in one way or another, I came across PyDev - Python support for Eclipse.

Preparing PyDev

+ Install Eclipse (I tend to grab the latest from Eclipse directly, since the repo copy is typically out-of-date, and it comes with its own updating functionality anyway).
+ Add PyDev's update site http://pydev.org/updates to Eclipse (Help -> Install New Software -> Add)
+ Install PyDev

A Few Prerequisites

It took me a good hour to get my project working properly with PyDev, so let me try to save you some of the hassle. 

1. Create a new project (PyDev -> PyDev Project)
2. Provide the correct project name (with the same spelling and capitalization as your source dir. If you change anything about it, you may run into difficulty having it locate your source files)
3. Configure your interpreter (autoconfigure should work ok)
4. Allow it to create your project. If your source files don't appear, go back to step 2 and check that project name and path carefully.
5. Go to project properties-> PyDev - PYTHONPATH -> External Libraries, and add the <projectname>_lib source folder
6. Window -> Preferences -> PyDev -> Interpreter - Python
  a) Under Libraries, add the <projectname>_lib folder
  b) Under Libraries, add the <projectname> folder
  c) Under Libraries, add the <projectname>'s parent folder

So, e.g. if your project is called foobar, you might have the following: /path/to/foobar/foorbar_lib, /path/to/foobar/foobar, /path/to/foobar. Without this seemingly excessive arrangement, PyDev won't be able to locate the nested imports under <projectname>_lib.__init__.py, and you'll see unreferenced import errors for the <projectname>_lib imports.

7.  Go to the next tab - Forced Builtins.
  a) Add "gi". This is required for PyDev to resolve the dynamically created references to the various GTK3 imports under gi.repository.XXX, such as Gtk, Gdk, GObject and Gio. If you look at /usr/lib/python2.7/dist-packages/gi/repository/__init__.py, you'll see what I'm talking about.
  b) Add "<projectname>". This is required for PyDev to resolve imports internal to your project. Otherwise, you're likely to see "from <projectname> import <projectname>Window" give an Unresolved import error.
  c) Click Apply for PyDev to resolve those gi.repository imports.

You should be set up.

A Stack of Improvements, Fit for a Toolstack

Now you have:

+ inline errors and warnings
+ auto-build
+ auto-indentation 
+ true auto-complete (not just random-word-from-the-same-file suggestion)
+ code formatting & import cleanup
+ refactoring capabilities
+ clickthrough to source
+ testing framework
+ a python console
+ a file browser

and all the other niceties of Eclipse. Isn't that better?

Oh, yes... amidst all that, I forgot to mention. It also has tabs, syntax highlighting, find/replace and plugins. What a relief!

Now the toolstack consists of:

+ Quickly - for boilerplate code to help get new projects started quickly, along with easy integration with Ubuntu's PPAs
+ Glade - for the UI design
+ Eclipse with PyDev - for coding

This is a much more acceptable configuration in my eyes.

I know Canonical is trying to keep it simple for those new to development, but I'm unconvinced that gedit alone is the solution. It can be quite hard to write clean code quickly without at least some of the functionality of an IDE.

To give an example: when I imported the code I'd written in gedit into Eclipse, up popped a raft of warnings for unnecessary import declarations scattered throughout my classes. This might just be an example of my sheer incompetence, but I suspect I'm not the only one who refactors some classes, and omits to remove all the affected imports.

I think, at the very least, the Quickly workflow needs a more featureful default editor, and some mention of the procedure to migrate to an IDE, such as Eclipse with PyDev, or even Vim.

But, credit where credit's due... it does at least support easily switching to Vim, given that quickly edit respects the EDITOR and QUICKLY_EDITOR environment variables. That is one saving grace at least.

</rant><p>Happy hacking!</p>

Troubleshooting - Run/Debug doesn't work in PyDev

Initially, you'll probably only see the following in the Eclipse Console:

pydev debugger: starting
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject

This is easy to fix. Quickly invokes the main method explicitly from bin/<projectname> with:

import <projectname>
<projectname>.main()

Similarly, PyDev needs an invocation of main() - otherwise it's not going to do very much. Open __init__.py in your project and add this at the very end:

if name == "__main__":
    main()

Now when you run/debug it, it should show signs of life before dying with the schema error (see below for background on it). This too is easy to fix - open the run or debug configuration window, and switch to the Environment tab. Add an XDG_DATA_DIRS variable with the value:

/path/to/<projectname>/data:/usr/share

For example, if the root of your project is at /home/bob/src/myproj, the value would be:

/home/bob/src/myproj/data:/usr/share

Troubleshooting - Settings schema not installed

You see the following error when running quickly debug, or python bin/<projectname>, etc:

GLib-GIO-ERROR **: Settings schema XXX is not installed.

I posted a solution to this on AU; in short, set XDG_DATA_DIRS=$XDG_DATA_DIRS:data, assuming you're in your project's root dir. For example:

cd myproj
XDG_DATA_DIRS=$XDG_DATA_DIRS:data python bin/<projectname>

Note that you have to have used quickly run to generate the compiled pyc files, and the bin/<projectname> script, before you can do this!

Troubleshooting - Glade unable to configure label text for GtkComboButton

There are a couple such bugs in Glade, which I believe to be fixed already. The corresponding version may already be in the proposed repos, although I can't remember where I saw this.

These are easy to work around by opening Glade's XML definition at data/ui/<projectname>Window.ui and removing/modifying the extraneous "checkbutton" text. Save your Glade work first, then open the XML file and make the necessary changes, and finally restart Glade.

Troubleshooting - Python memory leaks

I have a cairo DrawingArea which I'm using to display an updating line graph. When I left my app running overnight, I found that Rss (resident pages) had shot through the roof. A great tool to track down memory leaks in Python is heapy. It's not the most intuitive tool around, but it's not awfully difficult to use either.

For a start, edit __init__.py in your project, and:

+ add import signal
+ add signal.signal(signal.SIGQUIT, start_pdb) to the main method, before the call to Gtk.main()
+ add the method declaration:

def start_pdb(signal, trace):
    import pdb
    pdb.set_trace()

Start your application directly with python instead of quickly, as described earlier in this ridiculously long post. Then, from another terminal, send it the QUIT signal to drop to a debugging prompt

kill -QUIT $(ps aux | grep python | grep <projectname> | awk '{ print $2 }')

From there, you can proceed to analyse the Python heap with heapy.

It was immediately apparent from heapy that my problem didn't lie in the Python heap. The Python heap was occupying just 18MB out of a total residency of over 1GB.

Remember that Python sits atop C. So we have actually got 2 heaps here: the C heap, and the Python heap. 

To inspect the C heap, I fired up the old favourite valgrind. Whilst it didn't provide conclusive proof, it seemed to indicate that whilst the GTK3 libs weren't 100% watertight, the big problem was actually with pangocairo - the font library I'm using to draw nice, smooth, legible text.

After I dramatically reduced the frequency of invoking key pangocairo methods, memory utilisation stabilised over prolonged periods. If I get time, I'll try to confirm the problem is indeed in pangocairo, and track where it's leaking. (Note that pangocairo is a very useful library, and I don't want to discourage you from using it by any means - just keep one eye on the system resources!)