Core 2 Duo CPU whine - workaround for Linux

Do you use Linux on a laptop and fear you're going to be driven totally insane by the high-pitched whining noise (squeal/screech) coming from your Core 2 Duo CPU's badly designed cpu packaging and voltage regulation? Like proximity to your laptop gives you tinea?

Me too. A workaround that'll cost you a little battery life (but probably not more than a few tens of minutes out of four-plus hour runtimes) is to add processor.max_cstate=3 to your grub command line. This turns off the problematic C4 power state of deep CPU sleep at reduced voltage, but otherwise leaves power management unaffected.

( For Ubuntu users, dpkg-reconfigure grub2 will offer to let you edit the kernel command line. )

Using PKCS#12 client certificates with PgJDBC

(This post is a reference copy of a mailing list post I made explaining how to use client certificates with PgJDBC if you wanted to be able to accept user-provided PKCS#12 files.)

Using IET (ISCSI Enterprise Target) for Windows Server Backup

Windows Server Backup in win2k8 server is fantastic - it's a consistent, snapshot-based automatic backup system capable of full disaster recovery / bare metal restore. I'm not a huge fan of much of the way the Windows servers work, but the backup setup is fantastic. With one wee flaw...

Manual backups may be made to a network share, or to a local volume then copied to a network share. Fuss free, but only with operator intervention.

Unfortunately, automatic scheduled backups require direct access to a drive, they won't work on a mounted NTFS volume or on a network share. This doesn't do me much good for disaster recovery, as even a USB2 or FireWire drive nearby has a good chance of being destroyed by anything that takes out my server. It rained (and hailed) in my server room last month, so I'm taking disaster recovery even more seriously, and a nearby HDD just isn't good enough.

I could run a FireWire 800 drive over cat5e to the near-site backup location, but that's surprisingly expensive to do, especially as I want redundant storage to protect against pesky HDD failures. I have a perfectly good Ethernet-connected Linux server with a 10TB RAID array running Bacula to back up everything else on thge network, and I'd prefer to just use it for Windows Server Backup too.

The solution: Win2k8 has a built-in iSCSI initiator. Simply turn the backup server into an iSCSI target, then use Windows 2008's built-in iSCSI initiator to connect to it so Windows Server Backup sees it as a local disk and can write backups to it. This turns out to be astonishingly easy, at least on an Ubuntu system.

Security notice

The following configuration does NOT authenticate the windows server to the iSCSI target via iSCSI mutual authentication, so it may be possible to trick the server into backing up onto a different server and "steal" the backup. It also passes the actual backup over the network in the clear, as it doesn't use IPSec. You may wish to address those limitations in your implementation.

It would be a very good idea to enable mutual authentication, but by time of writing I was unable to get it working. The win2k8 iSCSI initiator complained about secret length, even though the provided secret appeared to match its criteria and had been entered in the main part of the control panel where the mutual authentication secret is expected. Similarly, IPSec wouldn't be a bad idea to prevent your backups passing over the network in the clear.

Configuring the iSCSI target

First, install the ISCSI Enterprise Target software (IET):

apt-get install iscsitarget

Now provision a volume to export as a target. This may be a local raw disk or partition, a logical volume provided by LVM, or even a great honking file on one of your mounted file systems. I'm using LVM, so I'll just allocate a logical volume:

lvm lvcreate -n winimagebackup -L 300G backupvg 

There is no need to format the volume; Windows does that. Just export it via iSCSI by adding a suitable target entry to /etc/ietd.conf (it might be /etc/iet/ietd.conf on your system):

 Target iqn.2010-01.localnet.backup:winimagebackup
        Lun 0 Path=/dev/backup/winimagebackup_iscsi,Type=blockio
        Alias winimagebackup
        IncomingUser iqn.1991-05.com.microsoft:winhostname xxxx

See the comments in the default ietd.conf and the ietd.conf man page for details on this. In brief:

• Change "localnet.backup" to the reversed host and domain name of your target server's name (mine is called "backup.localnet"). 
• Change "IncomingUser" to the user name you want the Win2k8 server to have to give to be permitted to connect, and "xxxx" to the password you wish to require. By default a 2k8 box will give the above user name, with "winhostname" replaced with the win2k8 box's hostname.
• Set the path after "Path" to the location of your storage.
• If you're using a file, you may need to specify "fileio" instead of "blockio" as the Type.

Restart ietd, and you're ready to connect the 2k8 box.

Connecting 2k8 to the iSCSI Target

Connecting to the target from win2k8 is similarly trivial. In the iSCSI Target control panel, in the "discovery" tab enter the dns name or ip of the target. Do not configure authentication (unless you've deviated from the ietd.confabove), just accept the dialog.

The server should appear in "target portals" and no error should be displayed. If successful, go to the "targets" tab, where you should see a target named "winimagebackup". Click "Log on..." to connect to it. Check the option to restore the connection at boot-time. Under Advanced, Configure CHAP authentication, using the password given in ietd.conf for IncomingUser under the target winimagebackup. Do not enable mutual authentication*. Accept the dialog, and the status of the volume should change to "connected".

Configuring Windows Server Backup

You're now ready to use Windows Server Backup with the volume. You do not need to format it under the disk mmc snapin before use. Just fire up Windows Server Backup and click "Backup Schedule", then follow the prompts, picking the iSCSI target as the backup storage when prompted.

Ubuntu Lucid and LTSP thin clients - wow

For a while I've been suspecting that most people who do any Linux-based GUI develpoment neglect remote X11 and thin client considerations. Performance has been decreasing slowly and painfully, and more and more workaround have been needed to get systems to behave.

Well, no longer. The new Ubuntu release, lucid, with the latest gtk etc absolutely screams along. I'm seriously astonished that remote X11 can be this fast, given the round-trip-happy nature of the toolkits and the protocol's flaws. It's a real pleasure to use.

To everyone involved in gtk development - thank you!. Doubly so to those who looked into my bug reports on specific areas where gtk used excessive network round trips, particularly the Evolution compose window bug.

By the way, for those of you who deploy LTSP, another thing that'll make a huge difference to performance is to make sure your LTSP clients are on a private VLAN and then enable direct X11 communication between client and server with LDM_DIRECTX = Y in lts.conf. With this option you still use ssh to login and establish a session (so there's no godawful XDMCP to fight), but only the ssh login is tunneled and encrypted. During session setup, DISPLAY is redirected to point directly to the client's listening X server. This offers a huge performance boost, especially to slower/older clients without onboard hardware crypto engines. (Oddly, the 600MHz Via C3 boxes outperform the Intel Core 2 boxes when all X11 comms are encrypted).

Splash screens are a plague upon Linux systems

How many splash screen tools can you remember appearing in at least one released version of at least one distro?

Too many, I bet, and all of them buggy.

In recent Ubuntu alone both usplash and plymouth have come to plague users. These tools purport to make bootup "friendly", but in fact:

• Cover up important messages/warnings
• Make recovering from issues during boot well-nigh impossible
• Make boot-time fsck fragile and hard to interact with
• Interact poorly with graphics drivers (xorg or kms)
• Are much, much too hard to disable

Unfortunately, they don't even have a standard mechanism for disabling them. "nosplash"  on the kernel command line used to work most of the time, but Plymouth displays a splash screen if the sequence "splash" appears anywhere in the kernel command line - including mid-word. It'll merrily accept "nosplash" as a request for a splash screen. With plymouth you must instead omit "splash" from the kernel command line entirely - and woe betide you if something else you need happens to include those characters!
 
Even better, Plymouth can't be uninstalled without ripping the guts out of an Ubuntu lucid system completely. It's wired deep into the package dependency system.

Argh. I'm coming to dread distro upgrades because I have to learn how to get rid of the blasted splash screen all over again. If only they'd stay uninstalled...

Intel, where's my EFI Network/USB/FireWire Target Disk Mode?

I'm torn.

I really don't like working with Mac OS X (or Windows, for that matter - I'm a Linux user by preference) .... but I love some features of Apple's hardware. The build quality and disk bays of the Mac Pro, for example. But above all else, what I love and envy about macs is ... Target Disk Mode. It's a service tech and sysadmin's dream.

Intel likes to make a lot of fuss about all its fancy in-chipset managment features, yet it seems to lack that one most crucial and handy feature - a Target Disk Mode equivalent. C'mon Intel, you can do better than this! You can not only implement FireWire target disk, but Ethernet-based iSCSI Target Disk for true sysadmin heaven. For bonus points, add TPM support so only authorized service techs for the company can get in, and use the built-in network management features to let admins remote-reboot a machine into target mode.

Sadly, I suspect the reason we're not all using this is that the "good" (cough, cough) old PC BIOS is still malingering, and failing to decently give way to EFI/OpenFirmware/whatever like it should.

Use Linux software RAID? Schedule periodic RAID scrubbing or lose your data

This post is bought to you by the fun of unnecessary wasted time and work rebuilding a server after a double-disk RAID array failure. RAID scrubbing is essential - and is supported by Linux's software RAID, but not used without explicit user action.

Linux's `md' software RAID isn't unique in this, but as its use is so wide spread it's worth singling out. No matter what RAID card/driver/system you use, you need to do raid scrubbing. Sometimes RAID vendors call this a "patrol read", "verify" or "consistency check", but it's all generally the same thing.

Why you should scrub your RAID arrays regularly

I had a disk fail in the backup server (some time ago now). No hassle - replace it, trigger a rebuild, and off I go. Unfortunately, during the rebuild another disk was flagged as faulty, rendering the array useless as it had a half-rebuilt spare and a second failed drive.

You'd think the chances of this were pretty low, but the trouble is that the second failed drive will have developed just a couple of defective sectors (a SMART check confirms this) that weren't detected because those sectors weren't being read. Until the drive was sequentially read during the rebuild, that is.

To reduce the chance of this, you can periodically verify your arrays and if bad sectors are discovered, attempt to force them to be remapped (by rewriting them from redundant data) or failing that fail the drive. This will also detect any areas where different disks disagree on what the correct data is, helping you to catch corruption caused by failing hardware early.

Unfortunately, Linux's software RAID doesn't do this automatically.

A simple shell script like this, dropped in /etc/cron.weekly and flagged executable, will save you a LOT of hassle:

#!/bin/bash
for f in /sys/block/md? ; do 
    echo check > $f/md/sync_action
done

Make sure to TEST YOUR EMAIL NOTIFICATION from mdadm, too. If a drive fails and you never get notified, you're very likely to lose that data the moment anything else goes wrong.

Use S.M.A.R.T too

For extra protection from failure, install smartmontools and configure smartd to run regular "long" tests of all your RAID member disks, so you're more likely to discover failing disks early.

Unfortunately, many consumer-oriented disk firmwares lie to the host and try to cover up bad sectors and read errors - probably to reduce warranty costs. Manufacturer's disk tools tend to do the same thing. Some even seem to lie during S.M.A.R.T self-testing, re-allocating sectors as they find bad ones and then claiming that everything is fine. In fact, I've actually had a consumer SATA drive that can't even read sector 0 return PASSED when queried for a SMART general health check, though at least it failed an explicitly requested self-test.

My point is that SMART testing alone isn't sufficient to ensure your disks are trustworthy, you really need to use a redundant array with some kind of parity or data duplication and do proper RAID scrubbing. And, of course, good backups.

If you use "RAID friendly" disks (usually the same physical drive with honest firmware and a much bigger price tag) you shouldn't have as many issues with SMART self-tests.