Splash screens are a plague upon Linux systems

How many splash screen tools can you remember appearing in at least one released version of at least one distro?

Too many, I bet, and all of them buggy.

In recent Ubuntu alone both usplash and plymouth have come to plague users. These tools purport to make bootup "friendly", but in fact:

• Cover up important messages/warnings
• Make recovering from issues during boot well-nigh impossible
• Make boot-time fsck fragile and hard to interact with
• Interact poorly with graphics drivers (xorg or kms)
• Are much, much too hard to disable

Unfortunately, they don't even have a standard mechanism for disabling them. "nosplash"  on the kernel command line used to work most of the time, but Plymouth displays a splash screen if the sequence "splash" appears anywhere in the kernel command line - including mid-word. It'll merrily accept "nosplash" as a request for a splash screen. With plymouth you must instead omit "splash" from the kernel command line entirely - and woe betide you if something else you need happens to include those characters!
 
Even better, Plymouth can't be uninstalled without ripping the guts out of an Ubuntu lucid system completely. It's wired deep into the package dependency system.

Argh. I'm coming to dread distro upgrades because I have to learn how to get rid of the blasted splash screen all over again. If only they'd stay uninstalled...

Intel, where's my EFI Network/USB/FireWire Target Disk Mode?

I'm torn.

I really don't like working with Mac OS X (or Windows, for that matter - I'm a Linux user by preference) .... but I love some features of Apple's hardware. The build quality and disk bays of the Mac Pro, for example. But above all else, what I love and envy about macs is ... Target Disk Mode. It's a service tech and sysadmin's dream.

Intel likes to make a lot of fuss about all its fancy in-chipset managment features, yet it seems to lack that one most crucial and handy feature - a Target Disk Mode equivalent. C'mon Intel, you can do better than this! You can not only implement FireWire target disk, but Ethernet-based iSCSI Target Disk for true sysadmin heaven. For bonus points, add TPM support so only authorized service techs for the company can get in, and use the built-in network management features to let admins remote-reboot a machine into target mode.

Sadly, I suspect the reason we're not all using this is that the "good" (cough, cough) old PC BIOS is still malingering, and failing to decently give way to EFI/OpenFirmware/whatever like it should.

Use Linux software RAID? Schedule periodic RAID scrubbing or lose your data

This post is bought to you by the fun of unnecessary wasted time and work rebuilding a server after a double-disk RAID array failure. RAID scrubbing is essential - and is supported by Linux's software RAID, but not used without explicit user action.

Linux's `md' software RAID isn't unique in this, but as its use is so wide spread it's worth singling out. No matter what RAID card/driver/system you use, you need to do raid scrubbing. Sometimes RAID vendors call this a "patrol read", "verify" or "consistency check", but it's all generally the same thing.

Why you should scrub your RAID arrays regularly

I had a disk fail in the backup server (some time ago now). No hassle - replace it, trigger a rebuild, and off I go. Unfortunately, during the rebuild another disk was flagged as faulty, rendering the array useless as it had a half-rebuilt spare and a second failed drive.

You'd think the chances of this were pretty low, but the trouble is that the second failed drive will have developed just a couple of defective sectors (a SMART check confirms this) that weren't detected because those sectors weren't being read. Until the drive was sequentially read during the rebuild, that is.

To reduce the chance of this, you can periodically verify your arrays and if bad sectors are discovered, attempt to force them to be remapped (by rewriting them from redundant data) or failing that fail the drive. This will also detect any areas where different disks disagree on what the correct data is, helping you to catch corruption caused by failing hardware early.

Unfortunately, Linux's software RAID doesn't do this automatically.

A simple shell script like this, dropped in /etc/cron.weekly and flagged executable, will save you a LOT of hassle:

#!/bin/bash
for f in /sys/block/md? ; do 
    echo check > $f/md/sync_action
done

Make sure to TEST YOUR EMAIL NOTIFICATION from mdadm, too. If a drive fails and you never get notified, you're very likely to lose that data the moment anything else goes wrong.

Use S.M.A.R.T too

For extra protection from failure, install smartmontools and configure smartd to run regular "long" tests of all your RAID member disks, so you're more likely to discover failing disks early.

Unfortunately, many consumer-oriented disk firmwares lie to the host and try to cover up bad sectors and read errors - probably to reduce warranty costs. Manufacturer's disk tools tend to do the same thing. Some even seem to lie during S.M.A.R.T self-testing, re-allocating sectors as they find bad ones and then claiming that everything is fine. In fact, I've actually had a consumer SATA drive that can't even read sector 0 return PASSED when queried for a SMART general health check, though at least it failed an explicitly requested self-test.

My point is that SMART testing alone isn't sufficient to ensure your disks are trustworthy, you really need to use a redundant array with some kind of parity or data duplication and do proper RAID scrubbing. And, of course, good backups.

If you use "RAID friendly" disks (usually the same physical drive with honest firmware and a much bigger price tag) you shouldn't have as many issues with SMART self-tests.

Battery backed cache for Linux software raid (md / mdadm)?

Linux's software RAID implementation is absolutely wonderful. Sufficiently so that I no longer use hardware RAID controllers unless I need write caching for database workloads, in which case a battery backed cache is a necessity. I'm extremely thankful to those involved in its creation and maintenance.

Alas, when I do need write-through mode (write caching), I can't use mdadm software RAID. There's actually no technical reason hardware already on the market (like "RAM Drives") can't be used as write cache, it's just that the Linux `md' layer doesn't know how to do it.

I say "it's just that" in the same way that I "just" don't know how to fly a high performance fighter jet with my eyes closed. Being possible doesn't make it easy to implement or practical to implement in a safe, reliable and robust way.

This would be a really interesting project to tackle to bring software RAID truly on par with hardware RAID, but I can't help wondering if there's a reason nobody's already done it.


Are you wondering what write caching is, why write caching is so important for databases, or how on earth you can safely write-cache with software RAID? Read on...

LVM vs VSS - it's no contest

Richard Jones writes about Microsoft's Virtual Shadow Copy Service (VSS, not to be confused with Visual Source Safe), and laments the lack of any equivalent or good alternative on Linux servers.

I couldn't agree more, but there's more to it than what he focuses on. Application consistency is in fact the least of the problems.

APP CONSISTENCY

App consistency and pausing is missing in Linux, but is in principle not hard to introduce. It is something where D-BUS can be very useful, as it provides another piece of the puzzle in the form of inter-application and application<->system signaling. The trouble is that, like with most things in the Linux-related world, some kind of agreement needs to be reached by interested parties.

All it really takes for app consistency is two d-bus events:

1. "Prepare for snapshot - pause activity, fsync, etc."
2. "Snapshot taken/failed/cancelled, safe to resume"

... though there needs to be some aliveness-checking and time-limiting in place so that a broken application can't cause indefinite outages in other apps by not responding to a snapshot message.

Off the top of my head, products that'd benefit from this include Cyrus IMAPd, PostgreSQL, MySQL, any other database you care to name...

... IS ONLY A SMALL PART OF THE PUZZLE

As compared to VSS, using dm-snapshot (LVM snapshots) on Linux suffers from a number of significant deficiencies and problems:

• Snapshots require you to use LVM. LVM doesn't support write barriers, so file systems must use much slower full flushes instead.
• Snapshots are block-level not file-system-level, so the file system isn't aware of the snapshot being taken.
• Because snapshots are block-level and not filesystem-aware, the snapshot must track even low-level file system activity like file defragmentation, erasing free space, etc. This means they grow fast and have a higher impact on system I/O load.
• Accessing a snapshot requires mounting it manually to another location and faffing around with the changed paths. There's no way for an app to simply request that it sees a consistent point-in-time view of the FS instead of the current state, as in VSS. This is clumsy and painful especially for backups and the like.
• The file system being mounted has to be able to cope with being mounted read-only in crashed state - it can't do journal replay/recovery etc. LVM doesn't even give the *filesystem* a chance to make its self consistent before the snapshot. Some file systems, like XFS, offer manual pause/resume commands that may be issued before and after a snapshot is taken to work around this.
• Snapshots don't age out particularly gracefully. You have to guess how much space you'll need to store changes, as LVM isn't smart enough to just use free space in the VG until it runs out. The snapshot's change tracking space is reserved and can't be used for anything else (even another snapshot) until the snapshot is freed. If you over-estimate you can have fewer snapshots and need to keep more free space around. If you under-estimate your snapshot may die before you finish using it. Argh!

So: even with the ability to pause app activity, there's unfortunately a lot more to be done at the LVM level before anything even half as good as VSS is possible. LVM has been progressing little, if at all, for quite a few years now and some of these problems (such as the namespace issues and lack of FS integration) aren't even practical to solve with a block-driver level approach like LVM's.

At this point, one can only hope that BTRFS can do a better job, so that we can switch to runnning btrfs on mdadm raid volumes and breathe a sigh of relief.

The Australian Maginot Line - because it worked great last time

Peter Thrush of ICANN recently commented that the Australian Internet Filter proposal is akin to the Maginot Line of WWII French fame. We all know how well that worked.

This is a surprisingly good analogy. The Maginot line presumed that the attacker would do what was expected of them, and wouldn't take the defenses into consideration when planning what they were doing. In much the same way, the Australian internet filter presumes that if it blocks what people do now, they won't change their behavior to circumvent the blocking with trivially available tools and techniques like encryption, tunneling, outside proxies, etc.

We already know that's an invalid assumption - not only is it rather contrary to general human nature, but it's being seen over and over in China with the Great Firewall. This despite the fact that China's Great Firewall is much more restrictive than Australia's is ever likely to be even under the most moralistic, conservative, idiotic government. Let's not forget, also, that in China it can be unhealthy to circumvent blocks that prevent you from accessing or posting information that's not meant to get around ... something I don't see becoming the case here.

So - in much more hostile circumstances, people still just waltz through the Great Firewall. Heck, I've done it myself - I had a workmate in China who needed unfiltered access, and it was the work of a few seconds to help him set up an encrypted SSH tunnel to a proxy on work's servers from which he could get to whatever websites he liked and do so undetectably. It's not even possible to tell that the encrypted data is web browsing data rather than something else.

Once again, it's clear that the only way the internet filter can work is if it's a whitelist. If a site isn't approved, you can't access it. If a protocol can't be inspected and content-filtered, it's blocked. No encryption of any sort may be used. Even that's imperfect due to cracking of whitelisted sites and use of them for proxies, etc.

It's a dumb idea. Why are we still wasting time and taxpayer money on such blithering idiocy?

Scripts to add replaygain tags to a flac collection

If, like me, you like to be able to play your music without constantly having to lunge for the volume control, and you store the master copy of your (ripped) CD collection in FLAC format you might be interested in a quick way to (re)calculate all the ReplyGain tags on your music. I'll also go into how to automatically create smaller MP3 copies of the files with the replaygain bias "burned in" so stupid players still get the volume right.
Finally, there's another little script here that can be used to fix up issues like inconsistent artist naming. It only does "The X" -> "X, The" at the moment, but it's easy enough to extend.
Read on if you're happy with shell scripting (and a Linux or advanced Mac OS X user).

The Great Australian Firewall just won't work

The proposed Australian Internet censorship rules will not work. Like most such blacklist-based schemes without active human monitoring it can be trivially bypassed by anybody capable of following simple instructions. As such, all the people it's designed to stop (like kiddie porn scum) will ignore its supposed blocking effects completely. Meanwhile we'll all have to live with the performance degradation, reliability problems, and so on.

Reasons why Microsoft Access absolutely stinks as a DB UI RAD tool

When used as a UI builder & RAD tool for accessing a real database (in this case PostgreSQL), Microsoft Access absolutely stinks. It has some severe problems that require cumbersome workarounds. In fact, if you're vaguely competent with a programming language it's almost certainly better to just write the application in your preferred language and be done with it. Java + Swing + Hibernate, for example, would do the job very well, as would C++ & Qt .
I was forced to use Access in this most recent project, and thought I'd post a warning for others who might be considering it but have alternatives available.
Note that none of this applies if you use MS SQL Server as the backend since Access doesn't use ODBC linked tables for MS SQL Server, but rather its own custom interface that's much smarter and more capable.
Problems using Access with ODBC linked tables include:

• It doesn't understand server-side generated synthetic primary keys. It can't just insert a record with a DEFAULT key and then retrieve the generated key through the ODBC driver. To use a PostgreSQL SERIAL column (a sequence) you must write an event procedure in VB that fires on the Form_BeforeInsert event. This procedure must issue a passthrough query to invoke nextval('seqname'), then store the value in the primary key field of the form. Hooray for a tool that does everything for you transparently, simply, and automatically.
• When you write your passthrough query, you will notice when examining the server query logs that it fires twice. That's because, if the ReturnsRecords property is set on a query, Access appears to issue the query twice. Presumably it's using the first instance to enumerate the returned fields for type-checking purposes, as it does not do this when the ReturnsRecords property is false. In any case, this means that you can't get the return value of a stored procedure with side effects without invoking the procedure twice. So, for nextval, I've had to first call nextval('seqname') and discard the return value, then call currval('seqname') to get the return value. That's two round trips to the database that were completely unnecessary, and a total of four for a single INSERT.
• Access has no concept of data dependencies between fields. If one field is (say) a combo box populated by a query that depends on the value of another field, you must use an event procedure in VB to trigger a refresh of the dependent field when the field it depends on is changed. This would seem to be one of the more obvious and basic things a RAD database tool would do for you, and I was stunned that it does not.
• Access loves to retrieve huge amounts of unnecessary data. If you open a form that's bound to a linked table, it'll issue a query like SELECT pkey FROM table;, which will result in a sequential scan on the whole table for databases that can't return the data from the index (like Pg¹). Even for databases that can, they must still read and scan the whole index. That's also potentially a lot of network I/O. You can work around this with EXTREMELY careful use of filter rules, but you have to be quite paranoid to make sure it never sees the form/table without a filter rule.
• Access cannot natively perform non-trivial queries server-side - you have to write them yourself with Visual Basic and use unbound forms, which isn't too dissimilar to how you'd be working with (say) Hibernate, just uglier. With a bound form and a linked table, Access will use a WHERE clause on the primary key when fetching a record, but that's about it. It can perform simple inner joins server side too, apparently, but I've hidden everything I can behind server-side updatable views so I haven't had to struggle with this aspect yet.
• Access cannot handle NULL boolean values.. Use a smallint with a constraint on it instead.
• Access needs some custom operators and casts defined to work with PostgreSQL.. It assumes that certain types may be cast to other types, compared to them, etc in ways that aren't allowed by default. Thankfully PostgreSQL supports user-defined operators and casts, so you can just add them. Here's one page with a useful set of operators and casts..
• Access likes to test for nullity with the = operator.. This is just wrong - the result of NULL = NULL is NULL, not TRUE, since the two unknown values may or may not be equal (the result is also unknown). PostgreSQL has a hack parameter, transform_null_equals, that may be set to enable the incorrect behaviour that MS Access expects. Access MAY have been fixed in version 2007; I haven't had cause to test this particular quirk yet.

You can work around a lot with careful use of filters, lots of Visual Basic code, plenty of views, updateable views, and stored procedures, and by writing your stored procedures to have EITHER side effects OR a return value (never both). It's not much fun, though, and it's still far from efficient.
On the upside, with row versioning enabled in the PostgreSQL ODBC driver, Access uses the MVCC attributes of the tuples (xmin, etc) to do optimistic locking. This allows it to avoid long-running transactions and long-held locks during user interaction without having to mess with the schema to add a field for oplocking use. It'll even play well with the oplocking used by things like Hibernate, and will work properly with apps that use ordinary transactional locking (SELECT ... FOR UPDATE and so on) without the need for any sort of support triggers. It's very nice.
To use Access with PostgreSQL, you REALLY need to set these options in the ODBC DSN for the connection:

• Row Versioning causes the driver to use only the primary key in WHERE clauses, and to use xmin & affected record count during updates to avoid conflicts.
• True as -1
• (unchecked) Bool as Char

¹ This isn't as stupid as it sounds. PostgreSQL uses an MVCC approach that allows for very efficient, highly concurrent inserts and selects. The price is that it must store tuple visibility information to know which transactions should see which tuples, since the table contains the data at many different points in time all at once. Visibility information isn't in the indexes (because it'd generate lots more write activity on the indexes, slowing down I/O) so once an index indicates a match on a tuple, it must be read from the heap to check visibility even if the index contains the actual data point we want. It's a performance trade-off that is not particularly great when you're doing a `count(*)' or `select pkey from table' but is wonderful on highly concurrent systems.

Nice work, NetworkManager

I'm yet to encounter a cellular modem that NetworkManager 0.7 (in Ubuntu 8.10 and 9.04-beta) doesn't automatically recognize without any user configuration, driver installation, or anything. Just plug it in (if not built in) and start using it.

Very nice work.

Dongles are EVIL

The device you see on the right is actually the devil. Or, at least, it's close enough if you are a system administrator.

It is a single piece of hardware that controls your access to business-critical programs. Lost the dongle? Whoops, no classified ads in the newspaper this week. Dongle broke? Ditto. Dongle fried by a computer malfunction or power fault? Ditto. Computer stolen? Ditto.

What's even more fun is that as computers move on and older interfaces become obsolete, it becomes hard to even find a computer you can plug the dongle in to. Most machines don't have parallel ports anymore, so parallel dongles like this one are a big problem. At least that can be worked around using USB adapters.

Of course, then you run into exciting issues like XP being unable to allow 16-bit code access to the parallel port. The program would work fine on XP, but for the stupid bloody dongle. So you're forced to maintain legacy hardware or waste time on complex emulation/virtualisation options just to get the program working, when it'd be just fine but for this dongle.

So, if you are ever offered software for any reason that requires a dongle, just say no.

Bought to you by the exciting battle to get an old and alas mission-critical win16 app to work under WinXP or even WINE.

Getting GNOME Evolution to offer a client certificate for IMAP SSL/TLS

GNOME Evolution isn't noted for its client certificate support. Entries in the bug tracker about it have rotted for years, and it has absolutely no acknowledged support whatsoever. Most other mail clients have had client cert support for years if not decades.

Unfortunately, Evolution is quite attractive in other ways - calendar integration, LDAP address books, etc. Unlike Thunderbird (especially when large images are involved) it also has acceptable performance over remote X11 connections.

So - I'd rather like to be able to use Evolution, but it's client support ... isn't.

It turns out, though, that Evolution uses the Network Security Services library from Netscape/Mozilla . It's used, among other things, for IMAP SSL/TLS support. This library does support client certificates; after all, Thunderbird and Firefox support client certificates and they do their crypto through NSS.

Is it not then possible to introduce a client certificate at the libnss level, so Evolution doesn't even know it's doing client certificate negotiation during its hand-off to NSS for SSL/TLS setup?

Why, yes, it is, and it takes one line of code in camel-tcp-stream-ssl.c to do it.

camel-tcp-stream-ssl.c:
- /*SSL_GetClientAuthDataHook (sslSocket, ssl_get_client_auth, (void *) certNickname);*/
+ SSL_GetClientAuthDataHook (ssl_fd, (SSLGetClientAuthData)&NSS_GetClientAuthData, NULL );

Because Evolution its self still has no idea about client certificates, if the server demands one and you don't have one installed you'll still get a useless error message instead of an appropraite prompt to install a client certificate. Just like Thunderbird and most other client-cert supporting apps. However, if you install a client cert by importing it into the Certificates section of the preferences, evolution (or more accurately libnss) will present it and use it when the server asks for it.


Update late 2009: 

Committed in stable (gnome 2.28.1+) http://git.gnome.org/cgit/evolution-data-server/commit/?h=gnome-2-28&id=87238717ceb0a158a00c76fc07c6e27c769c2cf0
Committed in master (gnome 2.29.1+) http://git.gnome.org/cgit/evolution-data-server/commit/?id=429a106d101bf205ba0c8ee8f94a818327c2d736

Update mid 2010:

This code has now hit shipping Evolution versions in up-to-date distros like Ubuntu 10.04 and Fedora 13. I've tested it in Ubuntu 10.04 and verified that client cert support works now. Hooray!

Getting central certificate management working on modern Linux

Modern Linux (GNOME, anyway) systems actually have a central certificate store. It's a bit lacking in management UI so far, but it works, and you can use it instead of loading your PKCS#12 certificates into every app you use manually.

First, import your certificate into the GNOME keyring with:

gnome-keyring import /path/to/certificate.p12

Install the libnss3-tools package (containing modutil).

Now exit every application you can, particularly your browser and mail client. Kill evolution-data-server too.

Find all instances of the nss security module database on your homedir, and for each one (a) test to make sure it's not open and (b) install the gnome-keyring PKCS#11 provider in it. The following shell script snippet will do this for you. Just copy and paste it onto your command line:

for f in $(find . -maxdepth 5  -name secmod.db -type f  2>/dev/null ); do
  echo "Testing: `basename $f`"
  if fuser `dirname $f`/cert8.db >&/dev/null; then
    echo -n "In use by: "; fuser `dirname $f`/cert8.db; echo " - Skipping"
  else
    modutil -force -dbdir `dirname $f` -add GnomeKeyring \
            -libfile /usr/lib/gnome-keyring/gnome-keyring-pkcs11.so
  fi
done

Now all your NSS-based apps should know about gnome-keyring and use the gnome-keyring certificate store.

If you use Evolution and want client certificate support, patch evolution-data-server as per GNOME bug 270893 to enable that too. It'll use gnome-keyring automatically.

i1Pro and the Australia Tax

The X-Rite i1Pro is an important instrument for anyone doing serious computer colour work, particularly for print and pre-press. It's also incredibly pricey, especially if you happen to live in Australia.

There's this oddity known locally as the "Australia Tax". The Australian Tax Office may not know about it, but it appears that local distributors for international businesses are convinced that it exists, and that it's high. That's the only explanation I can find for some of the jaw-dropping price differences between US and Australian versions of the same products - most of which are made in China anyway.

I got the X-Rite i1Pro instrument cheap (ish) - at AU$1500 ex GST and shipping compared to the AU$1800 quoted price. X-Rite force you to buy through exclusive local dealerships that add a huge markup, so while the US price is US$995 for the same instrument (AU$1200 @ current rates) you can't just order from the US. They won't ship it to you. You can use a US remailing service but X-Rite won't register it and won't support it outside the US - and neither will the AU distributor. You can't get it recalibrated etc without a painful amount of effort.

The AU dealership tries to claim it "adds value" ... but they don't do local advanced tech support, don't have any techs or offices outside metro Sydney, ship the instruments off to the US (3-4 week round trip) for calibration, and don't even keep spares in stock. So what value, exactly, do they add, other than to the price tag?

X-Rite and their distributors are raking it in with this arrangement. Unfortunately, X-Rite have been buying out all their competitors (like GretagMacbeth) so they're the only game in town. Like Quark, they'll suffer for their customer-hostile attitude and parallel import restrictions eventually, but right now they're in the "raking in the dough" phase.


(Of course, literally three days after I bought the i1Pro, Graham Gill, who develops Argyll CMS, announced support for the much cheaper ColorMunki spectrophotometer ... but hey, the i1Pro is a much better instrument so no harm done.)

Client certificate WTF

Why does almost nobody bother to support X.509 client certificates properly? They're a weak, poorly implemented afterthought in many systems if they're supported at all.
(Note: Android info here may be dated, as I last tested with 2.0, and applies only to the stock Android distribution not apps or patched versions supplied by vendors.)

• Microsoft Windows: Perfect support, though it requires PKCS#12 (.p12) format certificates. Most 3rd party apps (subversion, mozilla apps, etc) use own cert stores rather than the OS's for no good reason.
• Mac OS X: Limited support in OS keychain. No support in OS services like Apple Mail (IMAP+TLS, IMAPs, SMTP+TLS or SMTPs), WebDAV over HTTPs, etc so I have no idea why they even bothered adding support to keychain. Some apps have their own support, eg Mozilla apps via NSS, but OS has none and most Apple apps have none. No 3rd party apps seem to look in the system keystore for certificates.
• Linux systems: All major SSL/TLS libraries have support, but there's no system-wide or desktop-wide keystore or key management. Netscape security suite apps have good support but certificates must be installed individually in each app - even though the underlying libnss library supports  a shared user-account-level security store. GnuTLS- and OpenSSL-using apps must implement their own certificate management but can support client certificates if they provide appropriate callbacks. As a result, support is very app dependent and often buggy or very clumsy. Real-world support is inconsistent - eg Subversion supports client certs, but many svn front ends don't handle the cert prompts; Thunderbird & Firefox support client certs via NSS; Evolution supports via NSS but has broken nss init code that breaks client certificates and is often built with GnuTLS instead of NSS anyway; etc. Overall it's painful but usually usable.
• Symbian (Series 60) phones: Support is perfect in OS and apps. Very smooth.
• Sony Ericsson phones: Seem to have no concept of client certificates, and treat request for client cert by remote mail server as an SSL/TLS error.
• Windows Mobile phones: Basically perfect from all reports. Pity about the rest of the OS.
• Apple iPhone: Decent client cert store support. Unclear how much access 3rd party apps have. It's used for safari. Reported broken in Mail (comment by Martin).
• Android phones: are a near-total information void. Apparently it's just assumed you'll use Google's services, not (say) your own secure mail server with your work. Because, you know, who needs confidentiality anyway? If you download the SDK and phone emulator, you'll quickly find out that not only does the OS lack any way to import a client certificate or use one in negotiation, but it lacks any way to even import new CA certificates. That's stunningly, jaw-droppingly pathetic. Of course, this is a phone with a read-only IMAP client so it's not clear what, exactly, it's meant to do...

Sigh.

Update: It looks like there's half-baked and mostly user-inaccessible support for importing CA certificates in some flavours of Android. This app exploits the facility. Client certificates don't appear to be so blessed.

Google Android is not a smartphone OS

... it's a simple phone OS plus a web browser and some Google services. At least if the devkit phone simulator I used to see if this was something I might want to actually use is anything to go by.

It appears to lack some pretty fundamental things you'd expect from a smartphone.

• Ability to browse and view local files on phone memory or SD card, eg open HTML files, PDFs, etc
• An IMAP client that can delete messages, mark them as read on the server, etc
• Any ability at all to support corporate private CAs (Certificate Authorities), since it can't import new X.509 CA certificates
• Any X.509 client certificate support for secure mail and intranet access
• Decent sync and backup facilities to a laptop/PC. Oh, wait, you only use Google services, right?

This is Google's fancy new phone platform? Call me again in a few years, once you've grown up a bit - right now, even the iPhone OS is a more solid choice.

The best eBook reader for Linux is currently....

 Microsoft Reader run under WINE.

Sigh. Not only is it the best, it's practically the only one unless you're content with fixed-format PDF. Few eBooks are available in, or reasonably convertible to, HTML, and even if they were there aren't any HTML renderers that can do half-decent H&J. None at all can hyphenate even poorly, and justification support tends to be limited to clumsy expansion-only justification that is ugly and not very nice to read.

So, to get a decent result one would basically have to hand-convert a plain text or HTML format book (possibly after pdf-to-text conversion) to TeX and typeset it for a particular display. That's not exactly a nice, convenient way to sit down for a good read. Even then, unless you use pdftex and read with Adobe Reader it won't even remember your place!

By contrast the Microsoft Reader .lit format is fairly widespread, supports automatic and somewhat decent H&J (though nothing on TeX / InDesign ), remembers your place in each book in the library, tracks and manages the library without forcing a particular on-disk structure on it, supports easy drag-and-drop of a book onto the program even from Nautilus, etc. It's friggin' emulated* Windows software that hasn't been updated or improved since it was practically abandoned by MS in 2003 and it's still better than anything available natively for Linux.

The situation is just as dire for Linux-based ( and Symbian-based ) phones and tablets. Given the spread of Qt to more and more devices, as well as all major platforms, I'm increasingly tempted to start work on a Qt-based reader with decent H&J, library management, place tracking / bookmarking / margin notes, etc. But how can there not be something out there already? Am I just blind, or is there really a gaping hole this big in free software capabilities?

Any suggestions? Anyone interested in working on one?

* I know, I know; I just don't care that W.I.N.E. Kudos to the WINE folks for their amazing work.

Australian Bandwidth Pricing

I recently switched hosting of large files (12Mb or so PDFs) on my employer's website from the existing Australian host to one in the USA. Why? Because it's cheaper to send data from the USA to Australian users than it is to send it from within Australia..

About 100x cheaper, in this case, when comparing Anchor Networks per-Gb pricing to SimpleCDN's.

SimpleCDN doesn't have any Australian node(s). Data gets requested by their US nodes on first request, cached, and sent back to Australia. Yet they're incredibly, vastly cheaper than anything local I can find.

The root of the problem appears to be that Australian hosting providers charge all IP traffic as if it were to go via an international link. There's no provision made for peering or intra-national traffic at the majority of hosts. This may be an issue with the hosting provider its self, or it may be with their upstream bandwidth suppliers, but I don't care. Internet routing is designed to solve this sort of problem - thankyou BGP - and peering points exist for a reason.

It's actually way cheaper to store your data in the UK, Ireland, Germany, France, or pretty much anywhere except Australia even if your users are 100% Australian. Isn't that kind of sad?

Technical Support for Commercial / Proprietary Software

The Anchor Networks head sysadmin has an opinion on commercial support for software that's pretty similar to mine - it's garbage. Both of us have learned this from painful experience.

The post is well worth a read if you're in a sysadmin/tech line of work. It mirrors my experiences with several vendors very closely, except that this particular case doesn't include any inter-vendor buck-passing or blame games. There's a reason more and more of the systems at work run on software I have source code to and can rely on myself to maintain - because that way, things actually get fixed.

If you think Anchor's experience with dedicated commercial support organizations is bad, you should try contacting tech support for incredibly expensive commercial software you've licensed and asking them to support their product! I've had totally disinterested or completely useless support from vendors of ten thousand dollar software packages. After all, just because I paid for it doesn't mean I should expect it to work as advertised or expect them to be interested in fixing bugs, right?

Adobe, Quark, MYOB, Apple. This means you.

Anyway, the downside of doing all the support work in-house is that you need to have the skills to undestand and run the systems you use. You can't run a DNS server if you don't understand DNS, can't run a mail server if you don't understand IMAP,SMTP,POP3 and TLS, etc. However, given that vendor support seems to be totally useless except for problems a retarded monkey could figure out, it's beyond me how people with no understanding of the systems they work with ever get anything done, whether or not they're paying for support.

Maybe they don't? It'd explain a lot about many of the businesses I work with...