Friday, 31 January 2014

Migrating from Rhythmbox to Clementine

Some months back, an update rendered my main Rhythmbox installation largely unusable. Changing playlists would lock the application for about 60 seconds, and ultimately Rhythmbox would end up exhausting its stack and crashing. Not good.

To save myself from having to trawl the codebase searching for the needle in a haystack, I figured I'd switch to something shiny and fresh. After some searching, I settled on Clementine; a featureful, cross-platform player, boasting an Android remote control application, integration with countless online services, good performance with large libraries, visualisations, and the like.

Porting Metadata

In order to make the switch, I needed to migrate not only my songs, but also a few key pieces of metadata I decided I couldn't do without: rating, play count, and last played date.

Rhythmbox stores its library in an xml file, and Clementine stores its library in a sqlite database, and no migration tool existed that I could see. Hence, I hacked up a quick and nasty python script to do the dirty work and enrich Clementine's database with my additional metadata.

It relies on your having first imported your music into Clementine via the Clementine user interface.

The script searches Clementine's library, correlates songs (by path) between Clementine and Rhythmbox, and where it finds matches, it enriches Clementine's metadata.

The fact that it correlates songs by path location means that the location URIs in rhythmdb.xml must match the location URIs in Clementine's database; otherwise, the script won't be able to match them up.

If you're in the same boat I was, you may find this script useful to use as-is, or as a base to build on.

Forewarning

Caveat emptor: this script was tested once, using one library, on one system, with Rhythmbox 2.99.1, and Clementine 1.2.1. That is all.  Be forewarned: expect to have to delve into the Python code to get it working, or to have to restore your Clementine configuration from backup, or for your computer to spontaneously explode in disgust, or goodness knows what else.

This script was a quick hack for my one-time use, and is not something I've lovingly crafted and polished to make "production ready".

Method

First, prepare your system:
  1. Import your music library into Clementine via the Clementine user interface.
  2. If necessary, make a copy of your rhythmdb.xml library (by default, located at ~/.local/share/rhythmbox/rhythmdb.xml)and perform search and replace with the tool of your choice, to clean the location URIs of any symlinks (e.g, file:///symlink/Bach/1.flac should be resolved to file:///absolute/path/to/Bach/1.flac). If you haven't used any symlinks in paths to your music files, then skip this step.
  3. Take a backup of your Clementine configuration directory, just in case: cp -R ~/.config/Clementine ~/.config/Clementine-backup
Then, download the script and run it in dry-run mode:

$ wget http://content.nixnotes.co.uk/rhythmbox2clementine/rhythmbox2clementine
$ chmod +x rhythmbox2clementine
$ ./rhythmbox2clementine -d

If satisfied that things look sensible, run it in commit mode:

$ ./rhythmbox2clementine

If something went wrong and you need to restore your original Clementine configuration, simply restore the backup directory you made earlier:

$ mv ~/.config/Clementine-backup ~/.config/Clementine

Usage

$ ./rhythmbox2clementine -h
usage: rhythmbox2clementine [-h] [-d] [--rhythmdb RHYTHMDB]
                               [--clemdb CLEMDB]

Enriches a Clementine library with ratings and play counts from Rhythmbox

optional arguments:
  -h, --help           show this help message and exit
  -d, --dryrun         Don't actually commit any changes to the Clementine
                       database
  --rhythmdb RHYTHMDB  Override the default Rhythmbox library location
                       ~/.local/share/rhythmbox/rhythmdb.xml
  --clemdb CLEMDB      Override the default Clementine sqlite DB location
                       ~/.config/Clementine/clementine.db

Sample output

$ ./rhythmbox2clementine -d
Operating in DRY RUN mode
Using Rhythmbox XML DB path: /home/[USER]/.local/share/rhythmbox/rhythmdb.xml
Using Clementine sqlite DB path: /home/[USER]/.config/Clementine/clementine.db
Rhythmbox library contains 50529 songs
Clementine sqlite 3.7.17
Clementine library contains 50782 songs
DRY RUN: Not committing any changes to your Clementine library

8552 out of 50529 songs would have been enriched

Sunday, 27 October 2013

Configuring Logcheck on Ubuntu

System logs provide a wealth of important detail about activity on our systems; from incorrect configurations, to failing hardware, to remote attacks. If we don't check our logfiles, all this information goes to waste, and the only time we'll find out something is amiss is when scheduled events fail to run, or a drive fails, or the system is identified as part of a botnet...

That said, most of us have better things to do than trawl our plethora of overly verbose logfiles every few minutes. One way of meeting our systems halfway is to receive summaries of uncommon and important log entries on a regular basis. In my case, I have these summaries sent to a dedicated mail account, which I check via my phone, at my leisure.

This way, the most important information is collected and sent to me, for perusal when I have time. It's a win-win situation.

For the task of logfile distillation and email preparation, I use a handy tool called logcheck.
It contains a large collection of rules which filter out "uninteresting" log entries, leaving just the tastier morsels for my daily commuting pleasure. It also contains a set of rules for "violations"; significant events, which logcheck separates into another email to highlight their occurrence.

Over time, you'll find yourself tuning these rules to reduce your reading burden. With each rule being a simple regex string, this is no great ordeal.

Installing Logcheck

sudo apt-get install logcheck

Now to configure it:

sudo vim /etc/logcheck/logcheck.conf

The one change you do need to make is to tell logcheck where to send its reports to:

SENDMAILTO="recipient@domain.com"

Aside from that, it's worth reading over this file to see what else you'd care to change. I like to set the following; shortening the subject line and the message:

INTRO=0
DATE="$(date +'%H:%M')"
FQDN=0
ATTACKSUBJECT="Security Alert"
SECURITYSUBJECT="Security Event"
EVENTSSUBJECT=

Logcheck executes via cron, so there's no service to restart after updating the config file. By default, it runs 2 minutes past every hour.

You can force a report to run by invoking logcheck manually. Try it out now:

sudo -u logcheck logcheck

However, unless you're already running a mailserver, you're likely to find that nothing interesting happens at this stage. This is because logcheck expects to use the sendmail command to send the email message; and this is where things get interesting.

Sendmail

Sendmail has been in existence since the 1980s. It's a veritable swiss army knife of a mailserver. However, it's an interesting beast to try to configure.

Most end users, like myself, will simply want logcheck to send their status emails via an external mailserver - say, a gmail server, or one provided by their hosting company. Well, it should be easy enough to configure a relay to an external mailserver with a powerful tool like sendmail. But that wasn't my experience. Between a mess of dnl terminators, FEATUREs, bizarre config parameter names and the magical world of sendmail's string quotes, I started losing the will to live.

Switching from sendmail to postfix (a simpler, drop-in replacement) was a welcome improvement; but it was still too heavy a solution for a task as simple as this. Hash dbs, stunnel... it all involved an unnecessary number of moving parts. Ergo, for simple mail relay to an external mailserver, I sought an alternative.

Configuring Mail Relay with MSMTP

First, I tried ssmtp, which had a good reputation for being a simple, lightweight, and reliable solution for mailserver relay. However, that coredumped straightaway on my x64 box, so I switched to msmtp, which is very similar.

sudo apt-get install msmtp
sudo vim /etc/msmtprc

A simple configuration looks like:

defaults
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile /home/USERNAME/.msmtp.log

# Account fooBar
account fooBar
host smtp.foo.bar.com
port 587
# Note that this is simply envelope-from, not the mail's "From" header
from me@foo.bar.com
auth on
user me@foo.bar.com
password ySSwbUsnEz5M

# Use fooBar as the default account
account default : fooBar

msmtp will use STARTTLS where possible to enter TLS mode, so that passwords are sent over an encrypted channel. The documentation claims that it will not try to send the password in an insecure form over an insecure channel: "If you really want to risk your authentication data, you have to force msmtp to do that by manually setting the authentication method while TLS is off." If you like, you can confirm this via Wireshark or tcpdump. 
For simplicity here, I've hardcoded the password. Obviously, this is not good, and you'll want to take steps to remedy this once you have it working. A quick workaround involves limiting access to /etc/msmtprc to a particular group, e.g. "msmtp", and adding the logcheck user to that group. An arguably slightly more rugged approach involves using msmtp's passwordeval to invoke gpg (backed by gpg-agent to support non-interactivity) to decrypt a password file.
Either way, it's a very good idea to use a separate dedicated email account for logcheck notifications, as I have done; this way, there is much less to lose via password exposure (an attacker with local system access has the ability to view your logfiles anyway). 

Now, verify you can send a mail with:

echo "Test from msmtp" | msmtp recipient@domain.com

If you received that mail, congratulations. You've configured a command-line mailer. Now you need to override lsb-invalid-mta and tell your system to use msmtp as its sendmail-compliant mailer:

sudo ln -s /usr/bin/msmtp /usr/local/sbin/sendmail
sudo ln -s /usr/bin/msmtp /usr/local/bin/sendmail

Test with:

echo -e "Subject: I hope I receive this\nTest from sendmail" | sendmail recipient@domain.com

If you want, you can also configure it as your system's "mail" command mailer too. (It's not necessary for logcheck, but you may find it useful for command line e-mailing)

echo 'set sendmail="/usr/bin/msmtp"' | sudo tee -a /etc/.mailrc

Test with:

echo "Test from mail" | mail -s "This too" 
recipient@domain.com

So, in future, should you want to send mail from the command line, you can use either the "mail" or "sendmail" utilities.

Logcheck

With an external email relay configured, and the sendmail command working, logcheck should start sending its updates each hour. Try it out by creating a security event (try to sudo using an invalid password, for example).

Logcheck does have its flaws. Ironically, it doesn't offer much by way of logging itself, and its verbose mode (-dd) could provide more detail. Additionally, I'd like to see more configuration options than are on offer at present.

However, compared to letting your logfiles wrap around out-of-sight - and out-of-mind - this old tool still has much to offer for simple home system monitoring.

Happy log scouring!


Sunday, 16 June 2013

Sane Window Placement in Unity

I've never been terribly enamoured with Unity's window placement algorithm. Granted, it works reasonably well for single monitor configurations. However, once we start using larger multi-monitor configurations, windows all too often pop up on a monitor far away from the one presently in use. That's bad enough; but when we're talking about dialog windows too, it quickly becomes intolerable.

The simple but effective algorithm of "place beneath the pointer" is a tried-and-tested golden oldie. It is often found on the older-style window managers, built for productivity and simplicity (think descendents of fvwm). It also happens to be ideal for large multi-monitor configurations, as new windows are spawned where your pointer (representing your present view point) is.

Thankfully, switching to such a placement algorithm is made simple in Compiz.

Start "ccsm" via the Unity dash, locate the "Window Management" section, click on "Place Windows", and make the following changes:

Placement Mode -> Pointer
Multi Output Mode -> Use output device with pointer
Force Placement Windows -> enter the text "(type=Dialog) | type=Normal", without the quotes.