Truecrypt under FreeBSD

Truecrypt does work under FreeBSD, but it's not always as smooth an experience as on Linux.

A common problem is receiving an "Operation not permitted" error when trying to mount your volume. In my case, has almost always been attributable to an unclean filesystem.

1. How to get access to your data.

Mount the volume with read-only disabled, and with do-not-mount enabled. This gives you a decrypted virtual device under /dev. You can obtain the device path with:

# truecrypt -l

You're now in a position to run a fsck:

# fsck -t <filesystem> /dev/<device>

NB: If you can't remember the filesystem type, you may find testdisk's Analyze mode useful.

Assuming that the fsck completes successfully (probably returning ***** FILE SYSTEM MARKED CLEAN *****) you can remount normally:

# mount -t <filesystem> -w /dev/<device> /mnt

2. How to prevent it reoccurring.

Follow these three steps:

• Always unmount before rebooting/removing the media. 
• Ensure all unmounts are successful (e.g., they shouldn't fail with "Device busy").
• If you have Gamin running, then use a mountpoint under /media or /mnt

The third point is important. gam_server has a tendency to set up a large number of directory and file watches with nautilus. If you're stuck with umount returning the "Device busy" error, check what's preventing umount:

# lsof +D <mountpoint>
COMMAND     PID USER FD  TYPE DEVICE SIZE/OFF     NODE NAME
gam_serve 33424  ace 48r VDIR  0,176     1024        2 /export/d1
gam_serve 33424  ace 49r VDIR  0,176      512        3 /export/d1/.snap
gam_serve 33424  ace 50r VDIR  0,176      512 62507008 /export/d1/2011 

In such a case, Killing gam_server (Gamin) should work. The problem with this is that Gamin is restarted instantly once killed. And Gamin doesn't appear, at a glance, to be awfully configurable in terms of directory exclusions and suchlike.

Your best bet is to try to get a umount in just after killing the process:

# kill 33424 && umount <mountpoint>

That rarely works, since Gamin is restarted so quickly. But sometimes you can just beat it and get your umount in. Running a simple script, something like this (note that this hack only handles one PID), should manage to umount eventually. It's always worked in my case. It may take some time though - be prepared to let it run for a half hour or so before giving up.

# false || while [[ $? -ne 0 ]] ; do kill -9 $(lsof +D <mountpoint> 2>/dev/null | grep gam_serv | awk '{ print $2 }' | uniq) && umount <mountpoint>; done

So the moral of the story is, if you're running Gamin, mount to a mountpoint which is under one of Gamin's preconfigured exclude directories. And as far as I'm aware (without checking the source code), the only two preconfigured exclude directories are /media and /mnt.

The Best Mind Mapping Software on Linux

I generally dislike 'best of' posts, since it's rarely the case that the author has properly analyzed every alternative, or that a 'best' option can be determined objectively.

It's hard to argue that this is an exception. I haven't even tried any of the Linux mind-mapping alternatives; I have no desire to now. I just ask you to try this application. Try it. You'll see what I mean. It is just... superb. It is XMind.

XMind is clearly heavily based on the Eclipse codebase (I will forever be grateful to the Eclipse Foundation for the supremely liberal Eclipse license). As such, the GUI is highly customisable to a degree that would satisfy even the most hardcore KDE advocate.

In true Eclipse style, there is an outline pane, a properties pane, and the usual comprehensive key mapping configuration.

The maps produced with the default settings are indisputably aesthetically pleasing. Unfortunately, the "markers" (clipart graphics for use in maps) included with the software aren't quite to my taste, but are perfectly adequate nonetheless.

The real benefit comes when you really start using the software properly. You'll find that everything 'just works' in an intuitive manner. Dragging a subtopic away from a central topic 'breaks' the link and converts it into a floating topic. Add graphics to anything; attach files to anything. Need to break a subtopic out into more detail? Just hyperlink it to another sheet (or separate mindmap file, or anything else).

Provide easy navigation between associated maps with hyperlinked 'floating topics'. Add custom graphics to the marker list (native is 32px and PNG is supported. You can find some to import in the Open Icon Library). Import/export options are good, including support for other mind mapping software formats; so you can work with the output of popular Android applications, for example.

The base application is FOSS; dual-licensed with EPL and LGPL. However, a few of the lesser-used and more advanced pieces of functionality are both highly expensive and proprietary. This mightn't be enough to satisfy staunch FSF supporters, but I see it as a perfectly reasonable trade-off that has resulted in a very high quality application. I'd be interested to hear your thoughts.

For my personal mind-mapping requirements on the desktop, the free version of XMind is what I'll be using for the foreseeable future.

In case this post is irreparably tarnishing my reputation in the FOSS world, let me reassure you; for my wider diagramming needs, I do, of course, exclusively use Dia!

Quickly on Ubuntu: Making it Quicker with PyDev

I was pleased to see that the team at Canonical have been making some progress with their application development documentation and the Unity API.

There's clearly strong emphasis on helping beginners to programming get their hands dirty and hack some code. I think it's great to see - this is what *nix is all about.

Looking over things, the toolstack they're recommending contained no major surprises. Python is a popular scripting language with a solid range of libraries, which is nice and easy for beginners to pick up. And Canonical's made a tool called Quickly to help budding devs hit the ground running, by creating some of the tedious boilerplate around persisting preferences, creating key GUI windows, logging, etc. Quickly also handles some of the packaging, and can upload to a PPA.

I thought it would be fun to give the stack a test run.

But, alas, I had somehow missed one of the items on that toolstack page. I must have seen it, I suppose... I must have mentally filtered it out. Because it didn't seem possible. But when you type quickly edit to bring up the code, there it is, in the digital flesh.

The IDE Canonical is pushing, is... Gedit.

They can't be serious

But they are. The biggest argument I have with Canonical over their default toolstack is unquestionably the decision to set gedit as the editor. It's just a basic text editor. For any primitive GUI-based text editing needs I have, I generally use Leafpad or Gvim, which can get the job done quicker.

Gedit's biggest features are - get ready for them - tabs, syntax highlighting, find & replace (just about...), and plugins (of which there are some of note, but alas, too few). This may be enough to give Microsoft's good ol' Notepad application a good run for its money, but it falls short of what is expected in the 21st century for application development.

Come on Canonical. You must be jesting, right?

An Ode to Power
 
I'd personally be vastly more productive in vim than gedit, as would many experienced users. But the user base targeted by Quickly is unlikely to want to delve too deeply into NERD tree, especially after grappling with one of the first textual mode-based interfaces they'll have seen in a while! Let's face it; on the Windows side, Notepad++ is so vastly superior to gedit that it almost doesn't bear comparison, and yet few Windows devs would deign to use that as an IDE.

To make matters worse, quickly edit starts off by opening all the files in your project dir simultaneously. It doesn't take much coding before this becomes totally unwieldy.

I stuck it out for a while, but I had to find something better. After trying numerous Python IDEs in the repos and finding each to come up short in one way or another, I came across PyDev - Python support for Eclipse.

Preparing PyDev

+ Install Eclipse (I tend to grab the latest from Eclipse directly, since the repo copy is typically out-of-date, and it comes with its own updating functionality anyway).
+ Add PyDev's update site http://pydev.org/updates to Eclipse (Help -> Install New Software -> Add)
+ Install PyDev

A Few Prerequisites

It took me a good hour to get my project working properly with PyDev, so let me try to save you some of the hassle. 

1. Create a new project (PyDev -> PyDev Project)
2. Provide the correct project name (with the same spelling and capitalization as your source dir. If you change anything about it, you may run into difficulty having it locate your source files)
3. Configure your interpreter (autoconfigure should work ok)
4. Allow it to create your project. If your source files don't appear, go back to step 2 and check that project name and path carefully.
5. Go to project properties-> PyDev - PYTHONPATH -> External Libraries, and add the <projectname>_lib source folder
6. Window -> Preferences -> PyDev -> Interpreter - Python
  a) Under Libraries, add the <projectname>_lib folder
  b) Under Libraries, add the <projectname> folder
  c) Under Libraries, add the <projectname>'s parent folder

So, e.g. if your project is called foobar, you might have the following: /path/to/foobar/foorbar_lib, /path/to/foobar/foobar, /path/to/foobar. Without this seemingly excessive arrangement, PyDev won't be able to locate the nested imports under <projectname>_lib.__init__.py, and you'll see unreferenced import errors for the <projectname>_lib imports.

7.  Go to the next tab - Forced Builtins.
  a) Add "gi". This is required for PyDev to resolve the dynamically created references to the various GTK3 imports under gi.repository.XXX, such as Gtk, Gdk, GObject and Gio. If you look at /usr/lib/python2.7/dist-packages/gi/repository/__init__.py, you'll see what I'm talking about.
  b) Add "<projectname>". This is required for PyDev to resolve imports internal to your project. Otherwise, you're likely to see "from <projectname> import <projectname>Window" give an Unresolved import error.
  c) Click Apply for PyDev to resolve those gi.repository imports.

You should be set up.

A Stack of Improvements, Fit for a Toolstack

Now you have:

+ inline errors and warnings
+ auto-build
+ auto-indentation 
+ true auto-complete (not just random-word-from-the-same-file suggestion)
+ code formatting & import cleanup
+ refactoring capabilities
+ clickthrough to source
+ testing framework
+ a python console
+ a file browser

and all the other niceties of Eclipse. Isn't that better?

Oh, yes... amidst all that, I forgot to mention. It also has tabs, syntax highlighting, find/replace and plugins. What a relief!

Now the toolstack consists of:

+ Quickly - for boilerplate code to help get new projects started quickly, along with easy integration with Ubuntu's PPAs
+ Glade - for the UI design
+ Eclipse with PyDev - for coding

This is a much more acceptable configuration in my eyes.

I know Canonical is trying to keep it simple for those new to development, but I'm unconvinced that gedit alone is the solution. It can be quite hard to write clean code quickly without at least some of the functionality of an IDE.

To give an example: when I imported the code I'd written in gedit into Eclipse, up popped a raft of warnings for unnecessary import declarations scattered throughout my classes. This might just be an example of my sheer incompetence, but I suspect I'm not the only one who refactors some classes, and omits to remove all the affected imports.

I think, at the very least, the Quickly workflow needs a more featureful default editor, and some mention of the procedure to migrate to an IDE, such as Eclipse with PyDev, or even Vim.

But, credit where credit's due... it does at least support easily switching to Vim, given that quickly edit respects the EDITOR and QUICKLY_EDITOR environment variables. That is one saving grace at least.

</rant><p>Happy hacking!</p>

Troubleshooting - Run/Debug doesn't work in PyDev

Initially, you'll probably only see the following in the Eclipse Console:

pydev debugger: starting
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject

This is easy to fix. Quickly invokes the main method explicitly from bin/<projectname> with:

import <projectname>
<projectname>.main()

Similarly, PyDev needs an invocation of main() - otherwise it's not going to do very much. Open __init__.py in your project and add this at the very end:

if name == "__main__":
    main()

Now when you run/debug it, it should show signs of life before dying with the schema error (see below for background on it). This too is easy to fix - open the run or debug configuration window, and switch to the Environment tab. Add an XDG_DATA_DIRS variable with the value:

/path/to/<projectname>/data:/usr/share

For example, if the root of your project is at /home/bob/src/myproj, the value would be:

/home/bob/src/myproj/data:/usr/share

Troubleshooting - Settings schema not installed

You see the following error when running quickly debug, or python bin/<projectname>, etc:

GLib-GIO-ERROR **: Settings schema XXX is not installed.

I posted a solution to this on AU; in short, set XDG_DATA_DIRS=$XDG_DATA_DIRS:data, assuming you're in your project's root dir. For example:

cd myproj
XDG_DATA_DIRS=$XDG_DATA_DIRS:data python bin/<projectname>

Note that you have to have used quickly run to generate the compiled pyc files, and the bin/<projectname> script, before you can do this!

Troubleshooting - Glade unable to configure label text for GtkComboButton

There are a couple such bugs in Glade, which I believe to be fixed already. The corresponding version may already be in the proposed repos, although I can't remember where I saw this.

These are easy to work around by opening Glade's XML definition at data/ui/<projectname>Window.ui and removing/modifying the extraneous "checkbutton" text. Save your Glade work first, then open the XML file and make the necessary changes, and finally restart Glade.

Troubleshooting - Python memory leaks

I have a cairo DrawingArea which I'm using to display an updating line graph. When I left my app running overnight, I found that Rss (resident pages) had shot through the roof. A great tool to track down memory leaks in Python is heapy. It's not the most intuitive tool around, but it's not awfully difficult to use either.

For a start, edit __init__.py in your project, and:

+ add import signal
+ add signal.signal(signal.SIGQUIT, start_pdb) to the main method, before the call to Gtk.main()
+ add the method declaration:

def start_pdb(signal, trace):
    import pdb
    pdb.set_trace()

Start your application directly with python instead of quickly, as described earlier in this ridiculously long post. Then, from another terminal, send it the QUIT signal to drop to a debugging prompt

kill -QUIT $(ps aux | grep python | grep <projectname> | awk '{ print $2 }')

From there, you can proceed to analyse the Python heap with heapy.

It was immediately apparent from heapy that my problem didn't lie in the Python heap. The Python heap was occupying just 18MB out of a total residency of over 1GB.

Remember that Python sits atop C. So we have actually got 2 heaps here: the C heap, and the Python heap. 

To inspect the C heap, I fired up the old favourite valgrind. Whilst it didn't provide conclusive proof, it seemed to indicate that whilst the GTK3 libs weren't 100% watertight, the big problem was actually with pangocairo - the font library I'm using to draw nice, smooth, legible text.

After I dramatically reduced the frequency of invoking key pangocairo methods, memory utilisation stabilised over prolonged periods. If I get time, I'll try to confirm the problem is indeed in pangocairo, and track where it's leaking. (Note that pangocairo is a very useful library, and I don't want to discourage you from using it by any means - just keep one eye on the system resources!)