Sunday 27 October 2013

Configuring Logcheck on Ubuntu

System logs provide a wealth of important detail about activity on our systems; from incorrect configurations, to failing hardware, to remote attacks. If we don't check our logfiles, all this information goes to waste, and the only time we'll find out something is amiss is when scheduled events fail to run, or a drive fails, or the system is identified as part of a botnet...

That said, most of us have better things to do than trawl our plethora of overly verbose logfiles every few minutes. One way of meeting our systems halfway is to receive summaries of uncommon and important log entries on a regular basis. In my case, I have these summaries sent to a dedicated mail account, which I check via my phone, at my leisure.

This way, the most important information is collected and sent to me, for perusal when I have time. It's a win-win situation.

For the task of logfile distillation and email preparation, I use a handy tool called logcheck.
It contains a large collection of rules which filter out "uninteresting" log entries, leaving just the tastier morsels for my daily commuting pleasure. It also contains a set of rules for "violations"; significant events, which logcheck separates into another email to highlight their occurrence.

Over time, you'll find yourself tuning these rules to reduce your reading burden. With each rule being a simple regex string, this is no great ordeal.

Installing Logcheck

sudo apt-get install logcheck

Now to configure it:

sudo vim /etc/logcheck/logcheck.conf

The one change you do need to make is to tell logcheck where to send its reports to:

SENDMAILTO="recipient@domain.com"

Aside from that, it's worth reading over this file to see what else you'd care to change. I like to set the following; shortening the subject line and the message:

INTRO=0
DATE="$(date +'%H:%M')"
FQDN=0
ATTACKSUBJECT="Security Alert"
SECURITYSUBJECT="Security Event"
EVENTSSUBJECT=

Logcheck executes via cron, so there's no service to restart after updating the config file. By default, it runs 2 minutes past every hour.

You can force a report to run by invoking logcheck manually. Try it out now:

sudo -u logcheck logcheck

However, unless you're already running a mailserver, you're likely to find that nothing interesting happens at this stage. This is because logcheck expects to use the sendmail command to send the email message; and this is where things get interesting.

Sendmail

Sendmail has been in existence since the 1980s. It's a veritable swiss army knife of a mailserver. However, it's an interesting beast to try to configure.

Most end users, like myself, will simply want logcheck to send their status emails via an external mailserver - say, a gmail server, or one provided by their hosting company. Well, it should be easy enough to configure a relay to an external mailserver with a powerful tool like sendmail. But that wasn't my experience. Between a mess of dnl terminators, FEATUREs, bizarre config parameter names and the magical world of sendmail's string quotes, I started losing the will to live.

Switching from sendmail to postfix (a simpler, drop-in replacement) was a welcome improvement; but it was still too heavy a solution for a task as simple as this. Hash dbs, stunnel... it all involved an unnecessary number of moving parts. Ergo, for simple mail relay to an external mailserver, I sought an alternative.

Configuring Mail Relay with MSMTP

First, I tried ssmtp, which had a good reputation for being a simple, lightweight, and reliable solution for mailserver relay. However, that coredumped straightaway on my x64 box, so I switched to msmtp, which is very similar.

sudo apt-get install msmtp
sudo vim /etc/msmtprc

A simple configuration looks like:

defaults
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile /home/USERNAME/.msmtp.log

# Account fooBar
account fooBar
host smtp.foo.bar.com
port 587
# Note that this is simply envelope-from, not the mail's "From" header
from me@foo.bar.com
auth on
user me@foo.bar.com
password ySSwbUsnEz5M

# Use fooBar as the default account
account default : fooBar

msmtp will use STARTTLS where possible to enter TLS mode, so that passwords are sent over an encrypted channel. The documentation claims that it will not try to send the password in an insecure form over an insecure channel: "If you really want to risk your authentication data, you have to force msmtp to do that by manually setting the authentication method while TLS is off." If you like, you can confirm this via Wireshark or tcpdump. 
For simplicity here, I've hardcoded the password. Obviously, this is not good, and you'll want to take steps to remedy this once you have it working. A quick workaround involves limiting access to /etc/msmtprc to a particular group, e.g. "msmtp", and adding the logcheck user to that group. An arguably slightly more rugged approach involves using msmtp's passwordeval to invoke gpg (backed by gpg-agent to support non-interactivity) to decrypt a password file.
Either way, it's a very good idea to use a separate dedicated email account for logcheck notifications, as I have done; this way, there is much less to lose via password exposure (an attacker with local system access has the ability to view your logfiles anyway). 

Now, verify you can send a mail with:

echo "Test from msmtp" | msmtp recipient@domain.com

If you received that mail, congratulations. You've configured a command-line mailer. Now you need to override lsb-invalid-mta and tell your system to use msmtp as its sendmail-compliant mailer:

sudo ln -s /usr/bin/msmtp /usr/local/sbin/sendmail
sudo ln -s /usr/bin/msmtp /usr/local/bin/sendmail

Test with:

echo -e "Subject: I hope I receive this\nTest from sendmail" | sendmail recipient@domain.com

If you want, you can also configure it as your system's "mail" command mailer too. (It's not necessary for logcheck, but you may find it useful for command line e-mailing)

echo 'set sendmail="/usr/bin/msmtp"' | sudo tee -a /etc/.mailrc

Test with:

echo "Test from mail" | mail -s "This too" 
recipient@domain.com

So, in future, should you want to send mail from the command line, you can use either the "mail" or "sendmail" utilities.

Logcheck

With an external email relay configured, and the sendmail command working, logcheck should start sending its updates each hour. Try it out by creating a security event (try to sudo using an invalid password, for example).

Logcheck does have its flaws. Ironically, it doesn't offer much by way of logging itself, and its verbose mode (-dd) could provide more detail. Additionally, I'd like to see more configuration options than are on offer at present.

However, compared to letting your logfiles wrap around out-of-sight - and out-of-mind - this old tool still has much to offer for simple home system monitoring.

Happy log scouring!


Sunday 16 June 2013

Sane Window Placement in Unity

I've never been terribly enamoured with Unity's window placement algorithm. Granted, it works reasonably well for single monitor configurations. However, once we start using larger multi-monitor configurations, windows all too often pop up on a monitor far away from the one presently in use. That's bad enough; but when we're talking about dialog windows too, it quickly becomes intolerable.

The simple but effective algorithm of "place beneath the pointer" is a tried-and-tested golden oldie. It is often found on the older-style window managers, built for productivity and simplicity (think descendents of fvwm). It also happens to be ideal for large multi-monitor configurations, as new windows are spawned where your pointer (representing your present view point) is.

Thankfully, switching to such a placement algorithm is made simple in Compiz.

Start "ccsm" via the Unity dash, locate the "Window Management" section, click on "Place Windows", and make the following changes:

Placement Mode -> Pointer
Multi Output Mode -> Use output device with pointer
Force Placement Windows -> enter the text "(type=Dialog) | type=Normal", without the quotes.

Saturday 1 June 2013

Installing Linux on Samsung Laptops

I'd had enough of waiting for Samsung to comment on their laptop bricking fiasco. As you may recall from my earlier post, Samsung support had set out to obtain some information with good intentions, but ultimately could tell me nothing other than what I interpreted as Samsung refusing to honour their own warranty terms. I wasn't impressed.

Elsewhere in the ether of the Internet, I could see little sign of progress. In general, people are still waiting for Samsung to issue a statement. Perhaps one of their subsequent BIOS updates has already resolved the issue? They're not saying.

Having owned this Samsung NP350V5C-A0EUK laptop for 6 months now, I feel as though I've waited long enough. Now's time to bite the bullet and install Linux, regardless of the risks.

Needless to say, I can't be held responsible for any problems resulting from anyone choosing to follow the following instructions.

Due Diligence

Torvalds did pull a patch written by Matthew Garrett, which limits the amount of data written to UEFI storage, thus preventing (or at least reducing the likelihood) of the issue manifesting. To maximise my chances of success, I intended to install a distro of Linux which included this patch in its installation media.

To this end, I sought a recently released distro. An obvious candidate was Ubuntu 13.04, having being released just a month or so ago. According to the release notes, the installation media comes with Ubuntu kernel 3.8.0-19.29 based on Linux 3.8.8 (I also confirmed this on this directly on the installation media ubuntu-13.04-desktop-amd64.iso).

By comparing drivers/firmware/efivars.c in the diff with that in Matthew's patch, it was clear that the 13.04 installation media does indeed contain the patch - it's as safe as it's presently possible to get.

The other obvious thing to check was that my laptop's BIOS was up-to-date with the included SW Update application. It was.

Preparation

Frankly, there isn't much else to check. Ubuntu provides some general UEFI installation advice, but there isn't much new there.

I was going to leave things moreorless as they were out of the box - UEFI enabled, SecureBoot enabled - and set up a dual-boot with Windows 8. I would leave partitioning to be handled within Ubuntu's installer (you occasionally hear of concerns regarding shrinking Win 8 partitions using Linux tools without taking additional steps before the event - I didn't think such concerns were founded).

Installation

I used Startup Disk Creator on Ubuntu to create a bootable USB drive from ubuntu-13.04-desktop-amd64.iso. Then I booted it via Samsung's BIOS (entered using F2) via the "Boot Override" section in its "Exit" tab.

I chose to select "download updates whilst installing" to capture any further updates at the earliest opportunity which might possibly impact the stability of the UEFI (not that I thought there were any; but, caution never hurt). This had the effect of upgrading the kernel from 3.8.0-19 to 3.8.0-23 during the installation process.

Since I was essentially testing a default installation on Samsung UEFI laptops, I refrained from my usual custom partitioning and chose "Install Ubuntu alongside Windows 8", which I would imagine is the more commonplace installation choice. That also had the added benefit of saving me from accustomizing myself with the nightmare that is an OEM's Win 8 default partitioning scheme.

Post-Installation

Rebooting... worked. I was presented with a GRUB boot menu. Choosing Ubuntu booted into 13.04 successfully.
  • Graphically, Unity was working perfectly with the Intel HD4000
  • Win 8 partitions could be accessed with one click
  • Wireless was up - the network password had been remembered from when I entered it during the installation process
  • The battery meter was fine
  • Bluetooth was working
  • Suspend on lid close/restore on lid open was fine
Next up, booting back into Win 8. Choosing Windows 8 (loader) from the GRUB menu resulted in:

error: can't find command 'drivemap'.
error: invalid EFI file path.

Press any key to continue...

Booting to "Windows Boot Manager" via the BIOS' Boot Override menu didn't work either - it presented a Windows "your system has encountered an error" screen briefly, before rebooting. This wasn't good.

I continued booting to "Windows Boot Manager" via the BIOS. On the third attempt, Win 8's Automatic Repair (quite a nice feature, Microsoft - well done) kicked in and automatically started the boot repair process. No user input was required. It spent about 60 seconds "repairing", after which it rebooted.

"Windows Boot Manager" via the BIOS then succeeded - Win 8 was alive again.

As for the Windows 8 (loader) entry in the GRUB menu; this is just going to need some tweaking of the GRUB menuentry (some ideas can be found here). I did get it working directly via GRUB; I'll make the changes to the grub config later on.

Update:

Just comment out most of the grub menuentry for Windows 8; leave the second search line (search --fs-uuid --no-floppy --set=root <the_ID>) and replace the chainloader line with chainloader (${root})/EFI/Boot/bootx64.efi. Hopefully GRUB will have this remedied soon.

Conclusion

It worked close to flawlessly for me.

Whether that's because Samsung has fixed the issue in a BIOS update, or because of Matthew's kernel patch, or perhaps my particular laptop's firmware isn't affected, or just down to luck (and it might fail later on), I'm not sure. I certainly can't say it will work for you. All I can say is that, so far, it's worked for me.

If you do fancy installing Linux on your Samsung laptop, I'd advise a path similar to the one I've documented here. It's quick, easy, and intuitive - it is basically aligned with the default installation recommendations for 13.04 on UEFI.

So, the situation is looking promising. The real travesty is Samsung's continued, and continuing, blithely indifferent attitude towards Linux users. This appears to be a critical firmware bug which results in complete hardware failure. Failing to provide a fix in a timely manner, failing to provide updates to end-users, and refusing to fix machines which fail under warranty; that is not a consumer-friendly approach. Given that Samsung is now king of kings among Android mobile device manufacturers, their attitude is hard to understand, and even harder to accept.

Monday 29 April 2013

Samsung Laptops Still Can't Run Linux


Those keeping themselves apprised of Linux news will be aware of a particularly nasty issue that reared its head earlier this year. It was discovered that installing Linux could end up bricking some Samsung laptops.

Subsequently, Matthew Garrett discovered that it wasn't just Linux; Windows could also brick these laptops (although it seemed not to do so naturally).

So it came to be understood that the problem was a bug in Samsung's UEFI firmware, and might be something akin to a buffer overflow.

Linus et al were quick off the mark, removing the samsung-laptop kernel driver, which seemed most capable of triggering the bug. Of course, the new kernel version takes time to work its way into the distro's installation discs, but I would imagine it exists in Ubuntu 13.04 now.

Whether this change removes all possible triggers is another question. Nobody seems to know for sure. There are rumours that bricked systems may be recoverable by clearing the CMOS. But who really wants to risk it?

Like many people, I have a Samsung laptop on which I intend to install Linux, and had been following developments with interest. I kept expecting a BIOS update to emerge, yet nothing appeared forthcoming.

The end user community was still waiting; the Launchpad bug report was still open, and Ask Ubuntu's various questions were still open issues.

So, last month, having waited quite some time already, I sent Samsung support an email requesting a status update. Their response noted:

A BIOS update has been released for most systems that are effected by this problem. To see if there is a BIOS update this can be checked automatically by a program called SW Updater available on the support page (link below):
http://www.samsung.com/uk/support/model/NP350V5C-A0EUK-downloads?downloadName=FM

At that time (25th March), there was no BIOS update available, but a couple of weeks later BIOS update P08ABE became available in SW updater.

There was no description and no changelog. Furthermore, there was no information available about it anywhere online. Was this the update allured to in Samsung support's email?

I upgraded anyway, but without any confirmation that this fixed the problem, I turned to support a second time, requesting some more information on P08ABE. Their response was courteously worded, but wasn't what many of us was hoping to hear:
I am sorry that this information is not more readily available. I understand this is frustrating if you have been waiting to add Linux to
your machine.


Technical Support are only here for hardware faults and as such we only have access to the same information as that which is released to the public. As such I have no further information for you at this time.

I have approached Samsung regarding the issues surrounding the loading of Linux and the official response received was that we do not support the operating system. 


As such I am unable to help with the matter and can only advise that any problem caused by changes to the OEM operating system will not be repaired under warranty.
 

If there is anything else we can help with, please let us know.
I do appreciate support's frankness; when you're the bearer of bad news, it's often easier to try to gloss over things than it is to lay out the facts clearly and plainly.

To summarise:
  • No update for the public as of yet, and no timeframe at present
  • It's not yet known whether BIOS update P08ABE fixes the UEFI issue
  • Any problems caused by their serious firmware bug under Linux "will not be repaired under warranty" (although I'm not certain that's in keeping with their warranty policy)
Frankly, I did expect more from Samsung; I expected dev & test prioritisation, an unprompted flow of information to end users with an indication of a timescale, and an immediate acceptance that any firmware bugs would be covered by the warranty.

However, at this stage, things are still in flux; we'll have to wait to see where the pieces come to rest. The hope is just that they do so sooner rather than later.

Update: The support contact I'm in dialogue with (a most friendly and amiable chap) has offered to look into the P08ABE BIOS update. Hopefully I'll have some more information soon. I suspect that P08ABE won't be the fix we're waiting for, but it will be nice to have that clarity.

Update 2: Support has informed me that they are unable to provide any information about P08ABE.

Sunday 3 March 2013

Truecrypt under FreeBSD

Truecrypt does work under FreeBSD, but it's not always as smooth an experience as on Linux.

A common problem is receiving an "Operation not permitted" error when trying to mount your volume. In my case, has almost always been attributable to an unclean filesystem.

1. How to get access to your data.

Mount the volume with read-only disabled, and with do-not-mount enabled. This gives you a decrypted virtual device under /dev. You can obtain the device path with:

# truecrypt -l

You're now in a position to run a fsck:

# fsck -t <filesystem> /dev/<device>

NB: If you can't remember the filesystem type, you may find testdisk's Analyze mode useful.

Assuming that the fsck completes successfully (probably returning ***** FILE SYSTEM MARKED CLEAN *****) you can remount normally:

# mount -t <filesystem> -w /dev/<device> /mnt

2. How to prevent it reoccurring.

Follow these three steps:
  • Always unmount before rebooting/removing the media. 
  • Ensure all unmounts are successful (e.g., they shouldn't fail with "Device busy").
  • If you have Gamin running, then use a mountpoint under /media or /mnt
The third point is important. gam_server has a tendency to set up a large number of directory and file watches with nautilus. If you're stuck with umount returning the "Device busy" error, check what's preventing umount:

# lsof +D <mountpoint>
COMMAND     PID USER FD  TYPE DEVICE SIZE/OFF     NODE NAME
gam_serve 33424  ace 48r VDIR  0,176     1024        2 /export/d1
gam_serve 33424  ace 49r VDIR  0,176      512        3 /export/d1/.snap
gam_serve 33424  ace 50r VDIR  0,176      512 62507008 /export/d1/2011 

In such a case, Killing gam_server (Gamin) should work. The problem with this is that Gamin is restarted instantly once killed. And Gamin doesn't appear, at a glance, to be awfully configurable in terms of directory exclusions and suchlike.

Your best bet is to try to get a umount in just after killing the process:

# kill 33424 && umount <mountpoint>

That rarely works, since Gamin is restarted so quickly. But sometimes you can just beat it and get your umount in. Running a simple script, something like this (note that this hack only handles one PID), should manage to umount eventually. It's always worked in my case. It may take some time though - be prepared to let it run for a half hour or so before giving up.

# false || while [[ $? -ne 0 ]] ; do kill -9 $(lsof +D <mountpoint> 2>/dev/null | grep gam_serv | awk '{ print $2 }' | uniq) && umount <mountpoint>; done

So the moral of the story is, if you're running Gamin, mount to a mountpoint which is under one of Gamin's preconfigured exclude directories. And as far as I'm aware (without checking the source code), the only two preconfigured exclude directories are /media and /mnt.

Saturday 23 February 2013

The Best Mind Mapping Software on Linux

I generally dislike 'best of' posts, since it's rarely the case that the author has properly analyzed every alternative, or that a 'best' option can be determined objectively.

It's hard to argue that this is an exception. I haven't even tried any of the Linux mind-mapping alternatives; I have no desire to now. I just ask you to try this application. Try it. You'll see what I mean. It is just... superb. It is XMind.

XMind is clearly heavily based on the Eclipse codebase (I will forever be grateful to the Eclipse Foundation for the supremely liberal Eclipse license). As such, the GUI is highly customisable to a degree that would satisfy even the most hardcore KDE advocate.

In true Eclipse style, there is an outline pane, a properties pane, and the usual comprehensive key mapping configuration.

The maps produced with the default settings are indisputably aesthetically pleasing. Unfortunately, the "markers" (clipart graphics for use in maps) included with the software aren't quite to my taste, but are perfectly adequate nonetheless.

The real benefit comes when you really start using the software properly. You'll find that everything 'just works' in an intuitive manner. Dragging a subtopic away from a central topic 'breaks' the link and converts it into a floating topic. Add graphics to anything; attach files to anything. Need to break a subtopic out into more detail? Just hyperlink it to another sheet (or separate mindmap file, or anything else).

Provide easy navigation between associated maps with hyperlinked 'floating topics'. Add custom graphics to the marker list (native is 32px and PNG is supported. You can find some to import in the Open Icon Library). Import/export options are good, including support for other mind mapping software formats; so you can work with the output of popular Android applications, for example.

The base application is FOSS; dual-licensed with EPL and LGPL. However, a few of the lesser-used and more advanced pieces of functionality are both highly expensive and proprietary. This mightn't be enough to satisfy staunch FSF supporters, but I see it as a perfectly reasonable trade-off that has resulted in a very high quality application. I'd be interested to hear your thoughts.

For my personal mind-mapping requirements on the desktop, the free version of XMind is what I'll be using for the foreseeable future.

In case this post is irreparably tarnishing my reputation in the FOSS world, let me reassure you; for my wider diagramming needs, I do, of course, exclusively use Dia!

Sunday 13 January 2013

Quickly on Ubuntu: Making it Quicker with PyDev

I was pleased to see that the team at Canonical have been making some progress with their application development documentation and the Unity API.

There's clearly strong emphasis on helping beginners to programming get their hands dirty and hack some code. I think it's great to see - this is what *nix is all about.

Looking over things, the toolstack they're recommending contained no major surprises. Python is a popular scripting language with a solid range of libraries, which is nice and easy for beginners to pick up. And Canonical's made a tool called Quickly to help budding devs hit the ground running, by creating some of the tedious boilerplate around persisting preferences, creating key GUI windows, logging, etc. Quickly also handles some of the packaging, and can upload to a PPA.

I thought it would be fun to give the stack a test run.

But, alas, I had somehow missed one of the items on that toolstack page. I must have seen it, I suppose... I must have mentally filtered it out. Because it didn't seem possible. But when you type quickly edit to bring up the code, there it is, in the digital flesh.

The IDE Canonical is pushing, is... Gedit.

They can't be serious

But they are. The biggest argument I have with Canonical over their default toolstack is unquestionably the decision to set gedit as the editor. It's just a basic text editor. For any primitive GUI-based text editing needs I have, I generally use Leafpad or Gvim, which can get the job done quicker.

Gedit's biggest features are - get ready for them - tabs, syntax highlighting, find & replace (just about...), and plugins (of which there are some of note, but alas, too few). This may be enough to give Microsoft's good ol' Notepad application a good run for its money, but it falls short of what is expected in the 21st century for application development.

Come on Canonical. You must be jesting, right?

An Ode to Power
 
I'd personally be vastly more productive in vim than gedit, as would many experienced users. But the user base targeted by Quickly is unlikely to want to delve too deeply into NERD tree, especially after grappling with one of the first textual mode-based interfaces they'll have seen in a while! Let's face it; on the Windows side, Notepad++ is so vastly superior to gedit that it almost doesn't bear comparison, and yet few Windows devs would deign to use that as an IDE.

To make matters worse, quickly edit starts off by opening all the files in your project dir simultaneously. It doesn't take much coding before this becomes totally unwieldy.

I stuck it out for a while, but I had to find something better. After trying numerous Python IDEs in the repos and finding each to come up short in one way or another, I came across PyDev - Python support for Eclipse.

Preparing PyDev

+ Install Eclipse (I tend to grab the latest from Eclipse directly, since the repo copy is typically out-of-date, and it comes with its own updating functionality anyway).
+ Add PyDev's update site http://pydev.org/updates to Eclipse (Help -> Install New Software -> Add)
+ Install PyDev

A Few Prerequisites

It took me a good hour to get my project working properly with PyDev, so let me try to save you some of the hassle. 

1. Create a new project (PyDev -> PyDev Project)
2. Provide the correct project name (with the same spelling and capitalization as your source dir. If you change anything about it, you may run into difficulty having it locate your source files)
3. Configure your interpreter (autoconfigure should work ok)
4. Allow it to create your project. If your source files don't appear, go back to step 2 and check that project name and path carefully.
5. Go to project properties-> PyDev - PYTHONPATH -> External Libraries, and add the <projectname>_lib source folder
6. Window -> Preferences -> PyDev -> Interpreter - Python
  a) Under Libraries, add the <projectname>_lib folder
  b) Under Libraries, add the <projectname> folder
  c) Under Libraries, add the <projectname>'s parent folder

So, e.g. if your project is called foobar, you might have the following: /path/to/foobar/foorbar_lib, /path/to/foobar/foobar, /path/to/foobar. Without this seemingly excessive arrangement, PyDev won't be able to locate the nested imports under <projectname>_lib.__init__.py, and you'll see unreferenced import errors for the <projectname>_lib imports.

7.  Go to the next tab - Forced Builtins.
  a) Add "gi". This is required for PyDev to resolve the dynamically created references to the various GTK3 imports under gi.repository.XXX, such as Gtk, Gdk, GObject and Gio. If you look at /usr/lib/python2.7/dist-packages/gi/repository/__init__.py, you'll see what I'm talking about.
  b) Add "<projectname>". This is required for PyDev to resolve imports internal to your project. Otherwise, you're likely to see "from <projectname> import <projectname>Window" give an Unresolved import error.
  c) Click Apply for PyDev to resolve those gi.repository imports.

You should be set up.

A Stack of Improvements, Fit for a Toolstack

Now you have:

+ inline errors and warnings
+ auto-build
+ auto-indentation 
+ true auto-complete (not just random-word-from-the-same-file suggestion)
+ code formatting & import cleanup
+ refactoring capabilities
+ clickthrough to source
+ testing framework
+ a python console
+ a file browser

and all the other niceties of Eclipse. Isn't that better?

Oh, yes... amidst all that, I forgot to mention. It also has tabs, syntax highlighting, find/replace and plugins. What a relief!

Now the toolstack consists of:

+ Quickly - for boilerplate code to help get new projects started quickly, along with easy integration with Ubuntu's PPAs
+ Glade - for the UI design
+ Eclipse with PyDev - for coding

This is a much more acceptable configuration in my eyes.

I know Canonical is trying to keep it simple for those new to development, but I'm unconvinced that gedit alone is the solution. It can be quite hard to write clean code quickly without at least some of the functionality of an IDE.

To give an example: when I imported the code I'd written in gedit into Eclipse, up popped a raft of warnings for unnecessary import declarations scattered throughout my classes. This might just be an example of my sheer incompetence, but I suspect I'm not the only one who refactors some classes, and omits to remove all the affected imports.

I think, at the very least, the Quickly workflow needs a more featureful default editor, and some mention of the procedure to migrate to an IDE, such as Eclipse with PyDev, or even Vim.

But, credit where credit's due... it does at least support easily switching to Vim, given that quickly edit respects the EDITOR and QUICKLY_EDITOR environment variables. That is one saving grace at least.

</rant><p>Happy hacking!</p>

Troubleshooting - Run/Debug doesn't work in PyDev

Initially, you'll probably only see the following in the Eclipse Console:

pydev debugger: starting
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject


This is easy to fix. Quickly invokes the main method explicitly from bin/<projectname> with:

import <projectname>
<projectname>.main()

Similarly, PyDev needs an invocation of main() - otherwise it's not going to do very much. Open __init__.py in your project and add this at the very end:

if name == "__main__":
    main()

Now when you run/debug it, it should show signs of life before dying with the schema error (see below for background on it). This too is easy to fix - open the run or debug configuration window, and switch to the Environment tab. Add an XDG_DATA_DIRS variable with the value:

/path/to/<projectname>/data:/usr/share

For example, if the root of your project is at /home/bob/src/myproj, the value would be:

/home/bob/src/myproj/data:/usr/share

Troubleshooting - Settings schema not installed

You see the following error when running quickly debug, or python bin/<projectname>, etc:

GLib-GIO-ERROR **: Settings schema XXX is not installed.

I posted a solution to this on AU; in short, set XDG_DATA_DIRS=$XDG_DATA_DIRS:data, assuming you're in your project's root dir. For example:

cd myproj
XDG_DATA_DIRS=$XDG_DATA_DIRS:data python bin/<projectname>

Note that you have to have used quickly run to generate the compiled pyc files, and the bin/<projectname> script, before you can do this!

Troubleshooting - Glade unable to configure label text for GtkComboButton

There are a couple such bugs in Glade, which I believe to be fixed already. The corresponding version may already be in the proposed repos, although I can't remember where I saw this.

These are easy to work around by opening Glade's XML definition at data/ui/<projectname>Window.ui and removing/modifying the extraneous "checkbutton" text. Save your Glade work first, then open the XML file and make the necessary changes, and finally restart Glade.

Troubleshooting - Python memory leaks

I have a cairo DrawingArea which I'm using to display an updating line graph. When I left my app running overnight, I found that Rss (resident pages) had shot through the roof. A great tool to track down memory leaks in Python is heapy. It's not the most intuitive tool around, but it's not awfully difficult to use either.

For a start, edit __init__.py in your project, and:

+ add import signal
+ add signal.signal(signal.SIGQUIT, start_pdb) to the main method, before the call to Gtk.main()
+ add the method declaration:

def start_pdb(signal, trace):
    import pdb
    pdb.set_trace()


Start your application directly with python instead of quickly, as described earlier in this ridiculously long post. Then, from another terminal, send it the QUIT signal to drop to a debugging prompt

kill -QUIT $(ps aux | grep python | grep <projectname> | awk '{ print $2 }')

From there, you can proceed to analyse the Python heap with heapy.

It was immediately apparent from heapy that my problem didn't lie in the Python heap. The Python heap was occupying just 18MB out of a total residency of over 1GB.

Remember that Python sits atop C. So we have actually got 2 heaps here: the C heap, and the Python heap. 

To inspect the C heap, I fired up the old favourite valgrind. Whilst it didn't provide conclusive proof, it seemed to indicate that whilst the GTK3 libs weren't 100% watertight, the big problem was actually with pangocairo - the font library I'm using to draw nice, smooth, legible text.

After I dramatically reduced the frequency of invoking key pangocairo methods, memory utilisation stabilised over prolonged periods. If I get time, I'll try to confirm the problem is indeed in pangocairo, and track where it's leaking. (Note that pangocairo is a very useful library, and I don't want to discourage you from using it by any means - just keep one eye on the system resources!)