FreeBSD: TIMEOUTs under high I/O throughput

ace1 kernel: ad40: TIMEOUT - FLUSHCACHE48 retrying (1 retry left)
ace1 kernel: ata20: port is not ready (timeout 15000ms) tfd = 00000080
ace1 kernel: ata20: hardware reset timeout
ace1 kernel: unknown: TIMEOUT - FLUSHCACHE48 retrying (0 retries left)
ace1 kernel: unknown: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=3062893328


On a *nix fileserver purpose-built to host a 20TB zpool, this is not what you expect to see. Yet, I've been seeing a lot of this recently, and it always ends the worst possible way: in a hard kernel crash.

ZFS has performed its job admirably throughout, with no data corruption ever observed or recorded, but that doesn't make up for my fileserver crashing increasingly frequently. It used to be once a month or so; in the past week that worsened significantly to > once a day. Furthermore, the timeouts are highly dependent on I/O load: the more data I shift around simultaneously, the more likely I'm going to be holding the power button for 4 seconds.

Looking into it, I read that the timeout issue could be mitigated by patching the ATA driver to accept a configurable timeout value, and increasing it to a value which was non-problematic. Details can be accessed via the FreeBSD wiki's ATA page, under the heading ATA/SATA DMA Timeout Issues.

Whilst this may work (and kudos to Volker for testing it out), it didn't sound like an ideal solution; a bit like papering over the hole in the wall instead of re-plastering. Increasing a timeout doesn't address why the timeout is being hit in the first place.

The best solution seemed to be to upgrade from the pre-historic ATA driver to the newer, leaner AHCI driver, which I heard is far superior. Seen as that would require a kernel rebuild, it seemed like an excellent opportunity to upgrade FreeBSD from 8.2 (STABLE) to 9.0 (RELEASE). I spent the rest of the week backing up my ZFS pool again, just in case something inexplicable occurred...

As it transpired, the upgrade process was dead easy. I was running STABLE, so I needed to do a source upgrade. This may take a little longer than firing off freebsd-update, but it isn't a whole lot harder (just so long as you allow yourself enough time to sort out any pesky port update issues and to check over the changes to /etc). If anybody's interested I can post the upgrade procedure.

To cut a short story shorter, bringing the system back up with 9.0 was painless. I was a little concerned that the device renaming due to the AHCI driver migration (/dev/adXX devices changing to /dev/adaXX) might cause problems, but that too passed completely without incident, due to two little niceties:

1. ZFS doesn't rely on device names 
2. FreeBSD created symlinks to the new device names, meaning even my other filesystems mounted from fstab were unaffected

But the best bits were yet to come. All the old timeout problems have disappeared, leaving me with a rock-solid FreeBSD server (just as it should be). Furthermore, disk I/O throughput is noticeably improved. Now there's a handy bonus!

However, the mystery remains: what caused the ATA driver to shift all of a sudden, from logging occasional timeout errors, to consistently logging many & causing hard crashes multiple times per day?