Beware of soapy frogs

I've had it with HTC - thanks for the rescue, CyanogenMod + AAHK

2012-01-23T09:41:00.013+08:00

HTC pushed an Android 2.3.5 update to my Vodafone Australia-branded HTC Desire HD. There was no changelog, and along with the Android update it turns out I get a new version of HTC Sense (yay?) with all sorts of animations I can't turn off and extra bloat.

Great work HTC, you made the phone faster, then ruined it with more pointless animation. At least the "no window animations" setting used to work in the old version...

The new phone version is:

Model number: HTC Desire HD A9191
Android version: 2.3.5
HTC Sense version: 3.0
Software number: 3.12.178.2
Kernel version: 2.6.35.10-g931a37e htc-kernel@and18-2 #1 WEd Nov 9 14:04:03 CST 2011
Baseband version: 12.65.60.29U_26.14.04.28_M
Build number: 3.12.178.2 CL200874 release-keys

(from "settings->about phone")

I'd love to downgrade it (rooting it if necessary to do so) then reflash it with a sensible firmware from Cyanogen etc. Unfortunately, Vodafone/HTC seem to have broken existing root methods on the device with the latest update. It's not like this is my phone that I purchased outright or anything, so why should I be able to do anything with it? GRR!.

IMPORTANT: IF YOU ARE CONSIDERING ROOTING AND REFLASHING YOUR PHONE, UNDERSTAND THAT IT WILL PROBABLY VOID ALL WARRANTIES AND MAY DESTROY YOUR PHONE IF YOU MAKE A MISTAKE OR IT DOESN'T WORK PROPERLY WITH YOUR MODEL! This is not a guide, it's just a report of what worked for me.

psneuter reports:

$ /data/local/tmp/psneuter
/data/local/tmp/psneuter
Failed to set prot mask (Inappropriate ioctl for device)

Fre3vo reports:

$ ./data/local/tmp/fre3vo
./data/local/tmp/fre3vo
fre3vo by #teamwin
Please wait...
$

... never successfully getting root.

I've also tried the official HTC bootloader unlock tool for the Desire HD. It's been released for my Vodafone firmware version, so it should work, but it reports "unsupported firmware version". Sigh. Even if the bootloader unlock worked, I couldn't actually reflash the phone without temporary root access or a direct flashing tool that doesn't require using the phone to modify its own recovery partition.

So, what's a pissed-off phone owner to do?

Investigation suggests I may be able to directly reflash the device with tools called odin3 (a leaked Samsung android flashing tool) or the libusb-based Heimdall, but I'm having a hard time finding suitable images for the Desire HD, or much information about it. When I reboot my HD into what I think is download mode (bootloader menu -> recovery) heimdall can't see the device - whether I'm using the libusb drivers or the default drivers. Odin doesn't seem to see it either, and it looks like both are designed primarily for Samsung phones.

Stuck!

The long term solution is, as noted below, buy from a less closed and restrictive vendor. If I wanted a locked down device I would've bought an iPhone, it would've at least had more than a half day's battery life.

PS: The next person who posts a guide/walkthrough/howto without listing the version number of the latest firmware for which it is known to work is going to find themselves in a world of pain if I ever find them. The amount of stunningly bad writing on this topic is incredible.

UPDATE: A downgrade was successful with the AAHK (Advanced ACE Hack Kit), after which I could root the phone with fre3vo:

C:\Users\Craig\Downloads\Downgrade_v3\Downgrade>adb shell
$ /data/local/tmp/fre3vo -debug -start FBB00000 -end FFFFFFFF
/data/local/tmp/fre3vo -debug -start FBB00000 -end FFFFFFFF
fre3vo by #teamwin
Please wait...
Attempting to modify ro.secure property...
fb_fix_screeninfo:
  id: msmfb
  smem_start: 802160640
  smem_len: 3145728
  type: 0
  type_aux: 0
  visual: 2
  xpanstep: 0
  ypanstep: 1
  line_length: 1920
  mmio_start: 0
  accel: 0
fb_var_screeninfo:
  xres: 480
  yres: 800
  xres_virtual: 480
  yres_virtual: 1600
  xoffset: 0
  yoffset: 0
  bits_per_pixel: 32
  activate: 16
  height: 106
  width: 62
  rotate: 0
  grayscale: 0
  nonstd: 0
  accel_flags: 0
  pixclock: 0
  left_margin: 0
  right_margin: 0
  upper_margin: 0
  lower_margin: 0
  hsync_len: 0
  vsync_len: 0
  sync: 0
  vmode: 0
Buffer offset:      00000000
Buffer size:        8192
Scanning region fbb00000...
Potential exploit area found at address fbb7f800:1800.
Exploiting device...

C:\Users\Craig\Downloads\Downgrade_v3\Downgrade>adb shell
#

Unfortunately, ClockworkMod Recovery doesn't seem to work, though it's theoretically now flashed onto the phone. The phone still boots into regular Android recovery. I suspect that AAHK hasn't successfully unlocked the bootloader to S-OFF, probably because gfree seems to fail:

# ./gfree -f
./gfree -f
--secu_flag off set
--cid set. CID will be changed to: 11111111
--sim_unlock. SIMLOCK will be removed
Section header entry size: 40
Number of section headers: 44
Total section header table size: 1760
Section header file offset: 0x000138b4 (80052)
Section index for section name string table: 41
String table offset: 0x000136fb (79611)
Searching for .modinfo section...
 - Section[16]: .modinfo
 -- offset: 0x00000a14 (2580)
 -- size: 0x000000cc (204)
Kernel release: 2.6.35.10-gd2564fb
New .modinfo section size: 204
Attempting to power cycle eMMC... Failed.
Module failed to load: No such file or directory

UPDATE: It looks like the radio downgrade failed the first time around. Re-running AAHK successfully downgraded the radio and got me ClockworkMod recovery on reboot into recovery mode. Victory! CM install in progress.

UPDATE: CM 7 installed, but got into a reboot loop on startup. To fix this I rebooted back into clockworkmod recovery and used it to erase the cache and user data. A reboot then succeeded, dropping me into a clean Cyanogenmod install - without, of course, any of my data.

I then grabbed the latest Google Apps version, put it in the root of the SD card, and rebooted into recovery using Rom Manager's "reboot into recovery" mode, where I used "install zip from sdcard" to install the apps and did another "wipe data / factory reset".

After that, a reboot bought me into a Cyanogen environment with the Google apps, Market, etc. Phew!

Atmel ATtiny with built-in 433MHz (US: 310MHz) transmitter!

2012-01-20T10:50:00.006+08:00

While researching parts for my soil moisture sensors I stumbled across these awesome Atmel microcontrollers:

I was so excited I had to share. At about AU$8 each, these little beasties might make building wireless soil moisture sensors so much easier it's just not funny. The main problem is going to be ordering them, since Jaycar and Element14 don't carry them, and DigiKey has them as non-stock components with 4000 unit minimum volumes. They're 4-bit 8051-architecture micros so they're not going to be compatible with the ATmega or ATtiny range, so I lose the advantage of having the same arch on sensor and control system. For something as relatively simple as sampling an analog temperature and humidity sensor that may not be a big problem.

It may still land up being easier to use an ATtiny for the analog sensor controller and digital sensor data transmitter, so I can use (mostly) the same software tools as for the ATmega on the control board. I could then hook the sensor's ATtiny up to either some wiring for wired service, or to an RF transmitter IC for wireless operation without much if any change to the sensor codebase.

Atmel also have a family of RF receiver ICs (with matching tx modules or transceivers available) so I might be able to avoid the need for a breakout board / shield for the RF receiver support and just make it an optional component in the base design. Things like the ATA5723 /ATA5724/ATA5728 and the ATA5745 /ATA5746 RF receiver ICs could be awfully handy at about AU$4 each ... if I can find someone who'll sell them to me in less than 1,500 unit quantities. If not, there are lots of other highly integrated 433MHz RF receivers and transmitter ICs out there.

The ATA8204P3-TKQY looks particularly suitable; it's a slower and cheaper unit without UHF, but that shouldn't be a biggie for my use. It's cheaper than any of the other units except the ATA8202-PXQW 19 on digi-key, and should do the job fine. It's surface mount so it won't be assembly-friendly, though. An alternative might be the ALPHA-RX433S from RF-Solutions as that's packaged as a little module that'd be a bit saner to solder up.

DIY DC soil moisture sensor - early test successful

2012-01-17T12:45:00.008+08:00

On the the progressive difficulty scale of home built soil moisture sensors the bottom of the ladder is a DC soil conductivity sensor that uses simple resistivity measurement.

It took a couple of hours build one of those last night, most of which was spent incompetently attempting to produce a decent solder joint on steel wire and on the cleaned heads of galvanized nails. Anyone who can use a soldering iron without being a hazard to themselves and those around them should be able to whip something like this up in a few minutes.

First step: Simplest of the simple DC resistive sensor

After looking at the GardenBot design by Andrew Frueh, I landed up using something between his newer voltage-flipping design and his basic capacitor-based design - something that probably shares the faults of both, but was good enough for a quick and dirty experiment.

It's a simple circuit and it works, producing analog input values across a wide range that correlated well with soil moisture in my test flowerpot. It's also a stupid circuit that's really just a simple first-try toy, and isn't something I'd ever consider deploying in practice.

The 10k resistor is there for current limiting in case of a short; the 300kΩ to 3MΩ resistor is for you to tune the voltage range to fit your sensor probe length, spacing, etc so you get a good range of values. I landed up using a 630kΩ resistor for a pair of ~1mm steel wire probes about 10cm long. Your needs will vary.

If you don't want this simple circuit always-on (and always electrolysing - ouch!) you can use a digital pin in OUTPUT mode instead of the +3.3V rail, keeping it LOW except when you want to take a reading. Note that pin 13 has an LED and resistor inline, so you may want to use a different pin.

Next step: AC resistive sensor

I'm determined to avoid using two digital pins like Andrew has done for his bidirectional current sensor, so I need to produce low voltage AC. For the next version I'll be experimenting with generating a low voltage ultra-low-current AC square wave, probably using an astable multivibrator (also on Wikipedia) so I can synchronize my sensor reads to the oscillator easily.

Bizarrely it might be cheaper to do this with an IC containing an array of Schmitt triggers or with a 555 IC like the NE555 or TLC555CP (used like this) than with discrete transistor logic!

Either way, the oscillator will have to be embedded as local electronics near the sensor to avoid RF on the long cables out to the sensors, but that shouldn't be a big problem with simple circuitry like this.

Keep an eye out for the next post with work on AC/oscillator based resistive sensors.

https://github.com/ringerc/scrapcode/blob/master/arduino_avr/Arduino_Resistive_Soil_Moisture_Sensor_1.ino

Interested in soil moisture sensors and irrigation control? Start with the UF/IFAS virual extension series

2012-01-16T11:57:00.016+08:00

I've been having ... "fun" ... trying to find a way to build an affordable network of soil moisture sensors that don't require too much looking after.

It's harder than you'd think, but this UF/IFAS Virtual Extension series on soil moisture and irrigation has made it a lot easier to understand the different approaches and sensor types. It'll help you understand the differences between resistive and capacitive soil moisture measurement, introduce alternatives like tensiometers, etc. This is important whether you plan to DIY your sensors or buy off the shelf.

Unfortunately, the article series says very little about how to get your hands on working soil moisture sensors without your wallet screaming for mercy. I'm still working on that bit.

Promising options that need further investigation include:

Vegetronix (USA) VH400: Capacitive probe (?), US$35 each, shipping cost to non-USA addresses is extortionate. Not yet tested.
Honeywell HCH-1000 Hydrogrometer: At $5 to $7 ea, promising. Available cased or uncased. See HCH-1000 datasheet. They're even on pins for our soldering sanity. As these seem to be designed for air humidity I'm not sure they're going to be suitable, but at this price it's worth a go. To test.
DIY DC resistive soil moisture sensor: Sensitive to impurities in the water, like salt levels, and quite temperature sensitive. Prone to electrolysis causing corrosion of the electrodes. Tested successfully but too crude for real-world use.
DIY AC resistive soil moisture sensor: As the DC version, but less prone to corrosion in exchange for being a bit trickier to build (or using up extra I/O pins for the lazy-man's version). Yet to test.
DIY capacative soil moisture sensor: These are less sensitive to soil composition and water impurities, but appear to be very hard to get right. Yet to test.

Less promising or totally non-viable options include:

DIY neutron probe: Just kidding. Don't even think about it.
Senviro sensors. Senviro claims to be focused on low-cost sensors, but their website is useless. When I called them (Jan 2102) I was told that the company has been essentially mothballed for the last year and a half, though they're looking at picking it back up to focus on long range wireless sensors for park/golf course/industrial use, focused on high durability and long life at a price of "hundreds not thousands". Still not really in our range; not testing.
Holman WS7880 433MHz wireless soil moisture sensor: A Western Australian supplied sensor, presumably a rebadge of something made overseas. Available as sensor and base station bundle or as stand-alone sensor, since each base station supports 3 sensors. At AU$17 (Bunnings) these aren't cheap but they're not that expensive compared to many of the other options. They require near-line-of-sight and won't work through multiple walls (as specified, even through one). The sensors take 2xAA batteries. They're a big and white and might be a theft/vandalism risk, plus they're not that pretty, but at least they'll be hard to accidentally mow over if installed in lawn. I think they're using resistive metering. I bought a couple for testing and they work well when used as designed. Being able to use these from a microcontroller - either by repurposing the base station or by directly receiving the 433MHz RF signals from the base stations - would be exciting and fun, but is beyond the ability and resources I have at the moment without at least some documentation. I have some but won't be testing them with my microcontroller unless I can find out more about their radio protocol.
Honeywell HIH-6131 series Hydrogrometer (eg HIH-6131-021-001 @ AU$27 ea for 10+): i2c-enabled sensor with on-chip signalling, optional temperature sensor integration. See also this product page. I'm not sure even the filtered versions can cope with immersion, they're more designed for air humidity. Not that cheap ($28 and waaay up) or suitable, so untested. Pity, as i2c bus support would be great.
Honeywell HIH-4000/HIH-4100/HIH-4200/HIH-4201 series Hydrogrometers: Simple voltage-output sensor. Still not less than $25 ea in 10-unit quantities, filtered variants over $35. No tested.
HIH-4602: Thermisisistor. Over $100 ea, so totally unsuitable, not further investigated.
Sensirion SHT10/SHT11/SHT15/SHT71/SHT75: The "low cost" SHT10 is still over AU$60 anywhere I can find it, so I have no plans to test this for home irrigation control. Not tested.
Silicon Labs humidity sensors don't actually appear to exist. Their site lists the category but no datasheets, and none of their catalog vendors carry them. Not tested, possible vapourware.
Various scientific/industrial integrated soil moisture measurement systems: Seriously expensive, usually hundreds or thousands of dollars. When looking for sensors, this is mostly what you find. Obviously not tested.
Various Tensiometers: Not only are soil tensiometers expensive, but they don't work well in dry soil and require a lot of babying. They're totally unsuitable for irrigation control where they'll be left in place for long periods. Not tested.

I'm planning follow-up posts as I try different DIY meter designs. I'll include lists of articles I found during research and used as references, so hopefully it'll help others doing similar work down the track even if they all get given the big red X at the end of testing.

Using a RHT03 (aliases: RHT-22, DHT22, AM2302) temperature/humidity sensor from Arduino

2012-01-14T21:40:00.028+08:00

I picked up a nice compact little temperature and relative humidity sensor called the RHT03 for a project from Little Bird Electronics. It and very similar parts appear to go by the names RHT-22, DHT-22 and AM2302. You can find the part at SparkFun, Adafruit, etc too.

It took a lot more work to get it working than I expected, so I thought I'd write it up here for anyone else who is looking into it. There's sample code at the end of this post, but you should probably read the details because this is a quirky beast.

UPDATE: I since found a library on GitHub: nethoncho/Arduino-DHT22 that does a better job more simply and compactly. It works fine with my sensor. It needed some changes for Arduino 1.0 and some further tweaks to work how I wanted, so I've uploaded a fork here: https://github.com/ringerc/Arduino-DHT22.

Given that it's clearly a pretty commonplace part, I thought reading it couldn't be too tricky. Foolish me!Its datasheet almost in engrish and the sample code (warning: Word document) is pretty horrid. The sensor communicates over a custom (non-Dallas-compatible) 1-wire interface where the lengths of low and high pulses convey information. Parts of the datasheet are practically fiction.

Particular things to note when working with this sensor are:

When requesting a sensor read, you may have to actively pull the signal line up to +5V for 20-40μs after your initial 1 - 10ms low pulse.
The datasheet is a lie. In particular, instead of a neat 50μs low signal before sending a bit, my sensor may pull the line low for anything from 35 to 75μs. Similarly, 0-bit high pulses may be from 10-40μs instead of the documented 22-26μs, and 1-bit high pulses may be 60-85μs instead of 70μs.
Even when you're being tolerant of timing weirdness, you'll probably get the odd bad read. Adding a cap across the +5VDC input (as mentioned in the datasheet) doesn't seem to help, so it's probably not power fluctuation. Be prepared to re-try reads and remember to wait at least 2s between reads.
(Didn't trouble me, but): Remember to let the sensor settle for at least 2s after power-on before attempting a read, and at least 2s between reads.
Your code must be extremely fast to avoid missing state transitions when capturing the bit stream, whether you're using a looping or interrupt-driven capture method. I landed up buffering the stream to an array of pulse lengths, then interpreting it after the full stream was buffered. This would be less of an issue if the timings from the sensor weren't so erratic to start with.

Having finally got it working, I thought I'd post some sample code. It's very much that - sample code - not a library suitable for direct re-use. In particular, this code does not handle millisecond counter roll-over at all. Reads during which the millisecond counter rolls over will fail. That said, at least the checksum is verified and negative temperatures are handled correctly. (By the way, a USB cable coming out of the freezer door seal looks really weird).

You should really look at the GitHub project for DHT-22 linked to above rather than using this code.

To try the code with an Arduino Uno R3 (ATMEGA328 based), connect the sensor's I/O wire (pin 2) to digital pin 2 on the Arduino. Connect sensor pin 1 to +5V and sensor pin 3 and 4 to GND. Theoretically you should be able to let pin 3 float, but I've seen reports that some sensors have their internal ground on pin 3 instead of pin 4, so it won't hurt to ground both.

I use pin 2 on the Arduino because it's one of the two pins that support external interrupts on the basic Arduino boards. Pin 3 will also work, just change sensorPin in the example code.

You could make the code work without requiring interrupts, either by busy-waiting in a loop to detect level changes or by using the pulseIn function to read pulse lengths. I haven't implemented that, but it shouldn't be hard. You can re-use all the code except readSensor, which must be rewritten to capture the timings array a non-interrupt-driven way. The trick is making it fast enough.

https://github.com/ringerc/scrapcode/blob/master/arduino_avr/RHT03_DHT22_humidity_temp_sensor.c

Extending Arduino example CIRC-05 to use hardware SPI control

2012-01-12T21:50:00.012+08:00

For kicks, I've extended the basic Arduino shift register LED control example CIRC-05 from www.oomlout.com to use the Arduino's hardware SPI routines instead of software signalling.

As someone who has done very little with low-level electronics and who didn't know what SPI even was until today, this was embarrassingly easy. Kudos to the excellent Arduino libraries and the great documentation for making this simple.

I'm posting the re-written example for CIRC-05 here. It has the original software-based control as well as support for SPI, so you can see how similar the methods are.

(BTW, if you were wondering what a "latch" is in the IC, see this example.)

The only change required to the example circuit is that you must use pins 11 and 13 instead of pins 2 and 3, as the Arduino hardware SPI support has MOSI locked to pin 11 and CLK locked to pin 13. It also uses pin 12 for MISO, but we're not using that in this example.

Use output 11 where the wiring diagram says use output 2, and use output 13 instead of 3. No other changes are required.

The actual circuit on the breadboard is not altered in any way.

Here's the sketch code:

https://github.com/ringerc/scrapcode/blob/master/arduino_avr/Arduino_Tutorial_CIRC05_hardware_spi.c

SparkFun Inventors Kit CIRC-03 - Motor not working (spinning)? It's an error in the instructions

2012-01-12T10:39:00.006+08:00

I've started playing with some basic tutorial/toy electronics stuff using the Arduino platform and the "SparkFun Arduino Inventors' Kit" (hardly "inventors'", but anyway...) after picking it up as part of an order from the awesome outfit Little Bird Electronics. While generally good, I've hit an interesting issue with the kit that's worth documenting for anyone else who has it.

The short version: If you're using the SparkFun kit that specifies a 10kΩ resistor and the test circuit doesn't work (the motor won't spin) you might need to use a lower valued resistor between the transistor and pin 9 of the Arduino board.

If this is the case, you'll find that when you flick the motor's drive around with your fingers so it spins, sometimes it'll spin down slowly and sometimes it'll stop suddenly, depending on whether the Arduino is currently trying to drive it or not.

Check for all the usual errors before assuming the issue described here is what's wrong with your circuit. You might've reversed the flyback diode, made a poor connection on a power rail, etc etc.

The SparkFun kit for CIRC-03 ("Spin Motor Spin - Transitor & Motor") is based on Ardx CIRC03 but it has one difference that looks like it may be a significant error.

The SparkFun instructions specify a 10kΩ resistor but, at least with the motor supplied in my kit, the test circuit won't actually work with that much resistance between the P2N2222AG (actually marked P2N222A18 in my kit) transistor's base pin and pin 9 of the Arduino Uno board. Replacing the 10kΩ resistor with a 2.2kΩ resistor - like that specified by the ardx instructions - fixes the problem, allowing the transistor to pass enough current to let the motor spin.

A quick read suggests that P2N2222AG and P2N222218 are functionally equivalent, with different date codes or serial numbers, so that shouldn't be the cause of the issue. I'm not too sure on this point, though.

If you bought the kit and don't do any other electronics stuff you may not have a 2.2kΩ (red-red-red) resistor or anything vaguely similar lying around. If so, you can always connect up 6 (or so) of the 330Ω resistors supplied in the kit in series to achieve roughly the same resistance. Remember that to get them connected in series you must put the input of one and the output of another alone on a breadboard row. Connecting five or six of them will give you a sort of stair-step or saw-tooth pattern where you have resistors connecting, say, F14-G13, F13-G12, F12-G11, F11-G10, F10-G9 and F9-G8. You can then plug the lead from the transistor base pin into F8 and the lead from Arduino pin 9 into the other end of the resistor series at G14. If your resistors have long leads and you don't want to clip them you have to space everything out a bit more than this to avoid shorts across all those resistor wires; this is just an example of how it can work.

To make sure my motor wasn't just defective I bridged it across the +5V and -GND rails. This probably wasn't a smart thing to do without a resistor in series with the motor; I got away with it, but you might not.

Windows command line survival: findfiles, find+xargs's dim cousin

2011-12-11T19:10:00.006+08:00

Sometimes, we have to use the Microsoft Windows cmd.exe command line. Maybe it's at a client site or on a user machine where installing PowerShell, Cygwin or Mingw isn't appropriate. Maybe you're on a domain member so locked down by group policy you can't do anything if you want to. Either way, you have to use cmd.exe, and you're swearing.

I find out about the odd useful command that makes the Windows shell more usable, and I'll be trying to post them here. The first is forfiles, a basic alternative to find -exec or find|xargs.

If you're missing find -exec (or xargs) it turns out that Windows provides forfiles to do a limited subset of the work that find -exec can do. This'll help with simple jobs like flattening out a deep directory tree into a flat folder of files, running a command recursively on every file in a directory, etc.

For example, here's how to find every TrueType font file within the deep directory tree C:\deeptree and copy the files to C:\flat.

forfiles /s /p C:\deeptree /M *.ttf /c "cmd /c move @file C:\flat"

You really want to use an absolute path when you invoke move here, as it's invoked with the file's parent directory as its cwd, not the directory you ran the findfiles command in. Use a relative path and you'll just land up renaming all your files - which is a handy trick, but not what you intended.

In reality you'll also want to use more appropriate paths than folders off the root of C:\ . Beware spaces in pathnames, and remember that Windows has totally different rules to *nix in this area.

This particular case can be just as easily accomplished by doing a search for *.ttf from the parent directory, selecting all, copying, and pasting into an empty directory. More complex uses of this command like recursive renames or command invocations, not so much.

See forfiles /? for help.

Bulk conversion of OpenType (OTF) to TrueType (TTF) or Type 1(PFA/PFB) fonts using FontForge

2011-12-11T18:22:00.011+08:00

FontForge is a great tool for converting fonts. I needed to convert a lot of OpenType fonts to TrueType or Type 1 format to use them with Apache FOP, as FOP doesn't yet support OpenType fonts with CFF glyph formats. Doing the conversion in one FontForge script didn't work well, because (a) it leaked memory until it got OOM-killed and (b) it only used one CPU. So I wrote a helper shell script to control the job - posted here in case anyone else needs it later.

The script:

Runs multiple FontForge processes ($CPUS) for faster conversion;
Limits each FontForge process to converting 10 fonts before a new one is spawned, limiting memory leaks while avoiding too much launch overhead
Filters the conversion log to eliminate messages I'm not so interested in seeing. You can tune the filter (awk script) to your taste.

You can grab it here (raw text download)

PostgreSQL: Great even when you can see the warts

2011-12-07T13:41:00.006+08:00

I’m a very happy PostgreSQL user, and was interested to read a recent post by an MS SQL Server DBA advocating PostgreSQL.

This post is a response, enumerating areas where Pg may not be so hot after all. You might say it's my tendency to always see the negative, or perhaps it's just my desire to inform others of the downsides of something so the loud advocacy doesn't leave them feeling let-down when the reality hits. I'd like to think it's the latter. Telling someone how it is from the start is much better than showing them the rosy-goggles view until they're committed. So, from an enthusiastic PostgreSQL user who spends a lot of time helping out on the mailing lists, here's a list of some of the warts and quirks I'm aware of, so you don't have to find out about them the hard way:

This post is based on PostgreSQL 9.0, as I haven't really got into 9.1 yet. Each version improves a great deal, so don't be surprised if some of these issues are gone by 9.2.

Limited CPU concurrency. Pg can run many queries on many different CPUs, but one query can only run on one CPU at a time. If you have one really big, CPU-expensive query, Pg can’t complete it faster by using multiple CPUs. There are some limitations on concurrent disk use, too, but they’re much less significant. None of this affects workloads with larger numbers of smaller queries, where Pg easily maxes out the machine.
No query queueing or admission control (though connection pools help a lot). Pg can’t natively be told to “run no more than 10 queries concurrently, queuing connections in submission order”; every connection can be actively executing a query at the same time. Too many actively working connections lead to contention for resources, excess context switches, etc and slow overall throughput. Every connection has its own query executor, so even high (many hundreds) idle connection counts can be expensive in synchronization overhead and in RAM, plus they tend to lead to "stampeding herd" problems when the DB or app is restarted. Pg has a “sweet spot” for number of actively working connections that varies depending on the hardware, and it’s usually best to use a connection pool to queue queries and stop it exceeding that sweet spot. Good connection pools like PgPool-II and PgBouncer are available, so this is more of a wart than a real problem.
PostgreSQL has no true stored procedures and no autonomous transactions. PL/PgSQL functions always run within a transaction; they cannot commit that transaction or create others. (They *can* use savepoints via EXCEPTION blocks, though). dblink can be used as a workaround, but has its own issues. It looks like MS SQL Server may have the same limitation, but I’m not up to date with MS SQL Server and wanted to point this out in case it doesn’t.
All-or-nothing built-in replication. 3rd party replication options are not affected by this. You can’t currently control which databases or tables get replicated when you’re using the built-in wal-shipping or streaming replication, it’s all or nothing. This can be a pain if you have a busy but unimportant database and a less-busy but very important database, where replication requirements differ. You can work around it by using 3rd party replication tools. Alternately, you can run two PostgreSQL instances, but they can’t use the same shared memory segment so you’ll pay a RAM cost for it, plus they have to be on different ports or IPs.
No automatic savepoints. If one statement in a transaction fails, the whol transaction is automatically rolled back. You can’t pick it up again from the last successful statement unless you’ve been issuing SAVEPOINT statements before each statement that might fail. Savepoints have a performance cost, too, so this can slow things down as well as being a pain. In practice Pg’s behaviour is usually a good thing, but it’s occasonally a real pain.
Limited XML functionality. Support for in-db XSLT in particular is currently in an add-on module with known issues.
Limited performance monitoring and somewhat primitive performance monitoring tools.
work_mem is per-sort (etc) not per-connection, so it’s hard to give a query or set of queries the maximum possible RAM to work with w/o risking exceeding available resources or pushing too much cached disk data out. In practice this isn’t too big an issue, but it’s a bit ugly.
Incompatible upgrades. Until PostgreSQL 8.4, you used to have to do a full dump and reload whenever you wanted to do a major release upgrade, which was a frequently raised argument against PostgreSQL for larger sites. pg_upgrade fixes this for later releases, making upgrades much less painful, but major release upgrades still often contain gotchas or backward compatibility breaks that may require code changes to some applications. On the upside, changes are generally to make common mistakes more obvious or harder to make, fix bugs or design issues, and generally improve the system; they're not made casually. Backward-incompatible changes are also always documented in the release notes.

In other words: Pg isn’t perfect. It has quirks, and you should be aware of them. Despite that, it’s *REALLY* good for lots of workloads, and the price is hard to argue with. The generally really helpful and friendly community doesn’t hurt, either.

The very incomplete list of things I love about using Pg is:

Continuous improvement. Every major release adds something that's really handy for a project you're working on or fixes some frustrating limitation or quirk.
The community is friendly, open and very helpful. There are lots of people on the mailing lists and on Stack Overflow who're happy to help anyone who's made at least a minimal effort to explain their problem and their environment. Responses are usually detailed and helpful.
Devs are part of the community. Some of the core dev team are active on the user mailing lists - it's not uncommon to see a question posted, followed half an hour later by something like: "Yup, that's a bug; fixed in c6e3ac11b60ac4a8942ab964252d51c1c0bd8845 in head and backported to 9.0 and 8.4 for the next minor release". PostgreSQL is one of the only places where I've seen a bug found, reported, and fixed within half an hour.
User contribution, involvement and influence. Got a bug that really annoys you or a feature you really need? You can quite possibly fix it, interest someone else in fixing it, or pay someone else to fix it. I've written fixes for issues that annoy me and had them accepted, so I speak from experience. If you're not a coder, you can learn, or you can pay someone who already knows the codebase to work on an issue for you. Many major PostgreSQL features are developed through commercially sponsored work by consultants and the model appears to work well.
Support options. You have a choice in which organisations you use for support, ranging from "none, I'll do it myself" to a variety of consultants and companies. This means you're not stuck with a single complacent support provider who knows you can't go anywhere else if you don't like the price or the service. If you're not paying for support you can't demand anyone else fix an issue for you - but then when was the last time you had any luck getting Oracle or MS to fix a bug in software you were paying support for? With Pg, you can (via consultants or direct participation) get an important fix into the next patch release, and include it in your own builds until then so you have it immediately.
Extensibility. The ability to add your own data types, procedural languages, etc comes in very handy when you're doing more than writing accounting software.
Stability and a focus on correctness. PostgreSQL doesn't eat your data, and if it's not sure if something's safe it'll throw an error rather than going ahead and making it up as it goes along. This is really nice when you care about your data.
Great documentation makes a big difference to how nice Pg is to use day-to-day, as does the excellent psql command line client with its good error messages, good UI and built-in help.

No further work on Kobo

2011-07-27T09:25:00.004+08:00

I've stopped work on the Kobo software and dev env.

The Kobo is a fun platform but there isn't much point developing addons and plugins, because only a tiny proportion of Kobo owners will even realize it's a computer, let alone one they can enhance.

Most of the things I want to improve about the Kobo require access to the source code for the Nickel eReader application that runs on it. Kobo are not releasing this code - not even under NDA or other restrictive terms that permit fixes to be sent back to them without permitting distribution of the code to others.

The rest of the things I'd like to do need the QWS (Qt Window System) driver that's currently distributed only as a statically linked component of nickel. Without this I can't write my own interactive Kobo apps unless I hijack nickel's startup with a qt plugin hook. Doing it that way is ugly, inefficient, and prone to breakage, plus it renders the device useless as an eReader.

Given this, working on Kobo isn't going to be a productive use of my time. I learned a bit about ARM embedded systems and cross compiling, but I think I'll be leaving it at that.

Update to address Dan C's comments:

I'm not giving up on Kobo at all... just on developing fixes and enhancements for it.

As for the company's loss/missed opportunity: it's really not that simple.

Kobo's product isn't the hardware

Kobo's product isn't really the hardware, it's their software, brand, publisher relationships, website, and bookseller relationships. Their hardware is really generic - it's the same, or very nearly the same, as several other eReader devices and is made by a 3rd party.

The original Kobo was a netronix device, and I think the wifi is too. The Linux port and platform software on it was developed by Netronix, too, using an ARM SDK produced by CodeSourcery.

What makes the Kobo different is the Qt-based user interface Kobo inc built - "Nickel" - and its helpers. This software is the main differentiator they have. If they release it under a full OSS license, they enable competitors (other booksellers, other hardware makers, etc) to easily rebrand it and bundle it on their own generic eReader devices. Kobo is already fighting two juggernauts - Amazon and Apple - in this market, and will probably be up against Google soon too. They can't afford to enable small clone-competititors to use their stuff without contributing back.

GPLv3 isn't enough, either, because a rebrand could contribute and publish all their code but still direct sales to a different book store. There go the revenues Kobo relies on.

What they'd need to do is a "source available" release rather than an "open source" release as we know it these days. One where you can obtain the source, edit it, rebuild the app, etc, but cannot distribute it on your own devices, change the target bookstore, etc.

They've chosen not to do that. There may be good reasons why. Auditing sources for release to the public is expensive, even if it'll be released only under a restrictive license or even under an NDA. They'd also have to potentially fight enforcement actions against competitors who infringed despite the terms. Then there's the issue of DRM and Adobe Digital Editions, which quite likely has its own restrictive terms, probably including restrictions on release of sources for apps that use it.

Taking patches and dealing with random "developers" can be lots of work

Even if it were easy to release the sources, it's not clear that they'd see all that much benefit for the work. How much contribution would they really get? I speak from experience when I say that vetting and reviewing submitted patches is often harder than writing the code yourself. That's especially the case in environments people are less familiar with (like embedded linux developed using Linux desktops/workstations), projects with smaller communities, C++ projects using particular toolkits like Qt, etc.

Now add the inevitable group of enthusiastic-but-incompetent newbie developers who want you to hold their hand all the way. Mix in a few abrasive personalities who know they're always right, a few angry self-righteous demanders with a severe attitude of entitlement, and a couple of right idiots. You get the potential for a huge time sink that takes devs away from real work or requires dedicated people to pay to take care of it.

The plugin API was a non-starter

The alternative, of a plugin API, was pretty much a waste of time.

Writing a good plugin API is way harder than just opening up the code... which is plenty hard enough. You have to document the API, maintain compatibility across releases, keep the build system and SDK up to date, handle the additional tech support issues created by buggy plugins, etc etc. An app not originally designed for plugins requires *massive* work for plugins to be useful - I learned that one the hard way while working on Scribus.

Given that the Kobo is very much a consumer device where the vast majority of its users won't realize it is an embedded computer at all, let alone a hackable Linux-based one, and you get a whole lot of work for nearly zero return. Anyway, anyone likely to be building plugins is more likely to be writing smartphone or tablet apps.

So it's really not that simple

In their position, I would've wanted to release the code under NDA or a restrictive source-available license to interested devs who could show they already had a handle on the SDK and the platform. I'm not them, though, and I'm not in possession of all the info. They may've had good reasons for staying completely closed. Alternately, it's quite possible that the tech types wanted to open up but the lawyers/PHBs vetoed because of fear or misunderstanding. We'll never know.

At least if they'd opened up Nickel under a restrictive source-available license they'd get bug fixes contributed back. A plugin API would just be a time sink for them, to no benefit.

Java EE 6 - Traps, pitfalls and warts list

2011-07-14T08:07:00.018+08:00

Java EE 6 has the potential to be a great platform. It just needs plenty of cleanup, implementation bug fixing, and real-world use to get there.

One of the bigger issues is the significant array of spec inconsistencies and oversights that leave parts of the system working (or not working) in ways very different to one might expect. These issues can greatly increase debugging time and take important time away from productive development and into chasing issues in appservers, frameworks, and the specifications themselves.

I thought I'd make a list of some of the ones I've hit so far. I'm really hoping we'll see a Java EE 6.1 - a fixes-only revision that, unlike EE 7, focuses on polish and usability over adding new features. These are things I think are important to see covered by it.

The shortlist, explained in detail below:

You can't @Inject into a @FacesConverter. Use Seam 3 Faces to work around this and hope JSF 2.1 fixes it in Java EE core.

You can't @Inject into a JPA 2.0 EntityListener. Do a direct BeanManager lookup from JNDI or use Seam 3 Solder's BeanManagerAware to kind of hack around this; it's ugly. Lack of JPA 2.0 lifecycle hooks for EntityListener classes means Seam 3 Persistence can't fix this one.

Completely avoid all use of javax.annotation.ManagedBean. It was obsolete before release, is only inconsistently recognised, duplicates JSF2 functionality that's also obsoleted by CDI's @Named, and should simply be removed.

Weld (CDI) before version 1.1.1 is extremely buggy. If using Glassfish 3.0, 3.0.1, or 3.1 you need to update weld-osgi-bundle to 1.1.1. See these instructions for Seam 3.

Mapping of application-local resource-ref resource names in web.xml to a global resource name defined in glasfish-web.xml or jboss-web.xml doesn't work for persistence.xml jta datasource references and isn't supposed to work. Ignore all the examples and official documentation that suggest it should, they work with direct JDBC and with Spring but not with JPA 2.0.

If you encounter mysterious silent failures or null pointer exceptions, you probably forgot to include beans.xml. Your container will not warn you about this at deployment or runtime even if CDI annotations are present on classes.

Many of these workarounds require the use of Seam 3 (ie: "let's finish the work EE 6 started") modules. Because of Glassfish bugs, some additional work is required to use Seam 3 on Glassfish versions prior to the yet-to-be-released 3.2. See this previous post for some information.

In addition to the genuine issues with EE 6, there's also the learning curve to contend with. A while ago I wrote a broad conceptual overview of EE 6 intended for those coming to it cold, rather than migrating from EE 5 or from Seam. I felt it was necessary because of the struggle I had getting started with EE 6 and the bugs that bit me and confused me during the learning process. Perhaps it'll benefit others here or even provide insight into areas of the documentation and tutorial that would benefit from improvement.

Now, for the details:

CDI vs legacy

The Contexts and Dependency Injection (CDI, JSR299) spec was integrated into Java EE 6 quite late in the process. Most of the other specs were developed concurrently with little or no awareness of CDI and of how significant it'd be in Java EE 6. This shows in several areas that really impact usability and learning time:

JavaServer Faces 2 includes its own dependency injection framework, introduced in Java EE 6 alongside CDI. This uses the javax.faces.bean.ManagedBean annotation and the javax.faces.bean scope annotations @RequestScoped, @SessionScoped and @ApplicationScoped. These directly duplicate functionality in CDI but only work within JavaServer Faces, creating a fair bit of confusion.
CDI doesn't work everywhere, and fails silently where unavailable. Because it was introduced quite late, there are places like JPA 2.0 EntityListener classes, JSF 2.0 @FacesConverter classes, etc where injection is not performed. Because CDI never sees these classes, injection fails silently with no error message or other indication so the confused programmer just gets NullPointerExceptions on some injection sites. Needless to say, this doesn't aid learning Java EE 6. It wastes a lot of development time with finding workarounds and additional maintenance, too.
In areas where CDI is unavailable there is often no good workaround for obtaining access to application resources like (say) an @ApplicationScoped pool of converters for some resource. Some libraries, like JSF2, have lifecycle hooks that permit partial injection support - for example, Seam 3 Faces adds injection but not @PostConstruct support for @FacesConverter. For some other libraries, like JPA 2.0 implementations, there aren't any hooks to use, so user classes have to do their own direct bean lookups using BeanManager. Getting a reference to the BeanManager requires a use of an add-on library like Seam 3 Solder or lots of messing with JNDI lookups. Using the BeanManager once you have a reference to it is messy, poorly documented and inefficient. Seam 3 Solder's BeanManagerAware class (hack) helps a little, but is still a very ugly approach.
EJB 3.1 EJBs have their own annotations that look similar to CDIs but have very different semantics. The annotations @Singleton, @Stateful and @Stateless declare EJBs, which have container-managed transactions and locking, interface remoting, and more. They also have subtly different lifecycle and scope rules to those for CDI managed beans (@ApplicationScoped, @SessionScoped and @RequestScoped). The EJB vs CDI managed bean differences are quite difficult for the newcomer to understand, especially given the apparent paucity of side-by-side comparision documentation covering both CDI and EJB 3.1 together.
There are two copies of the ManagedBean annotation, because of an effort to "standardize" bean annotations that occurred parallel to CDI. The javax.annotation.ManagedBean copy is confusing, only works in some of the places where the javax.faces.bean copy does, and should never have been included in the JDK in the first place because it was already obsoleted by cdi by the time of release.

How should these issues be solved?

Move javax.faces.bean and javax.annotation.ManagedBean into a separate "JavaEE6-legacy" spec jar, and flag them @Deprecated. New projects should not see these annotations at all.
Make CDI work almost everywhere, and make the exceptions explicit. This requires spec and implementation changes in some places to define lifecycle hooks and hookable factories for helpers like FacesConverters and JPA 2.0 EntityListeners.
Where CDI isn't available for performance or other reasons, ensure that the CDI implementation is required to report a warning or error at deployment time if annotation scanning finds CDI annotations in classes where they cannot be effective. For example, an @Entity class annotated @RequestScoped or containing @Inject fields should be an error at deployment time. It won't always be possible to detect, but would be a good start.
Provide a civilized way to access and use the BeanManager from code that still can't get it any other way, via a standard BeanManagerLocator helper class and helper methods in BeanManager for common bean lookup tasks. Legacy code isn't going to go away in a hurry, and not all code can be written to run only within a container; some of it has to work outside a managed container environment too. CDI's purist approach has hurt usability here.
Make the mixing of CDI and EJB annotations an error at annotation processing time during deployment.

JSF2 and JAX-RS have incompatible access methods for HttpServletRequest etc

Another area where CDI's late introduction shows is in the completely different approach JAX-RS takes to injection. JAX-RS uses method argument injection, not instance variable injection. If you want access to (eg) HttpServletRequest in JAX-RS, you add a method parameter of type HttpServletRequest that's annotated @Context. That's all well and good if you're only writing JAX-RS code. But what if you want to write a class that's callable from JAX-RS and from JSF2 pages?

You have to write two different versions of each method. One that uses JAX-RS method argument injection to get the HttpServletRequest (or whatever else you need) and another that uses FacesContext to obtain it via getExternalContext().

You can't just @Inject HttpServletRequest req; as a member variable, because CDI injection doesn't know how to inject HttpServletRequest. You have to use framework-specific methods to obtain it.

Thankfully, the Seam 3 Servlet library largely resolves this issue by introducing a servlet filter that captures HttpServletRequest and stores it in a threadlocal exposed to injection via a CDI producer. This should become part of Java EE 6.1.

JNDI naming is confusing and hard to debug

Despite ongoing efforts into EJB 3.1 and Java EE 6, JNDI naming shows its messy app-server specific history strongly. Different app servers have different naming schemes. It's hard to get a clear picture from within an app of what the JNDI name tree contains from the app's perspective - which, with Java EE 6's resource-ref mapping and module/component scopes is now different to the app server's perspective.

A few changes would really help here:

Require everything within an app to perform lookups up the component->module->app->global scope hierarchy. Right now there are some exciting exceptions (like persistence.xml) that make the java ee 6 scopes much less useful, and much much more confusing, than they should be.
Define some static helper methods on JNDI contexts that recursively dumped a context to an outstream. Yes, they're fairly easy to write, but would be rather helpful for people trying to get an initial handle on issues, especially those who started with Java EE 6 who consequently have little direct contact with or experience with JNDI.
Define portable global JNDI naming for all resource types, not just EJB 3.1s, so app servers are guaranteed to offer a consistent name for a given resource in a given place.

Related to the above: persistence.xml ignores resource-ref mappings

An important specific case of JNDI issues is with persistence.xml. Different containers like to name resources differently and different deployments have different needs. That's why web.xml has the resource-ref element for declaring an app-internal name for a resource that must be defined, and why container-specific deployment descriptors have elements for mapping referenced resources to global jndi names in the container.

Unfortunately, persistence.xml cannot use those mappings. The persistence context is initialized outside module/component scope and must use global mappings. At least, that's what the Glassfish folks say:

The resource-ref element that you refer to above defines an indirection within a component name space. A PersistenceUnit (EMF) is not initialized within a component and hence always uses the global name space to look up data sources.

Again, we have a case where a useful feature in Java EE 6 only works in some places, some of the time. Again, I wasted hours and hours wondering what I was doing wrong, writing test cases and filing bugs before being told that, actually, it's not supposed to work.

What does work - in glassfish - is defining an app-local data source within web.xml using the <data-source/> element. This is, unlike defining data sources in glassfish-web.xml and jboss-web.xml, theoretically portable. For an example of a datasource defined this way, see this article. Define the datasource in the java:app/env namespace, eg as java:app/env/your-ds-name, and reference it from persistence.xml as java:app/env/your-ds-name. It works for me on Glassfish 3.2 build 09 ... but of course that means it has to trigger an apparent bug in JBoss AS 7.

Configuration and deployment descriptors are within the application archive

In Java EE 6, application configuration lives within the application .war / .ear, in xml deployment descriptors or sometimes even in annotations on the Java code its self. That's great for quick and easy testing, but severely limiting for production.

You have to unpack and edit a war to change deployment-specific details like:

principal-to-role/user mappings in glassfish-web.xml;
resource-ref mappings in (glassfish|jboss)-web.xml;
datasources defined in web.xml or in code using Java EE 6 standard annotations;
datasources, jms resources, javamail sessions, etc defined in glassfish-resources.xml;
... and more.

This makes it nearly impossible to distribute a signed war for any non-trivial app using standard EE 6 configuration mechanisms, because the app must be unpacked and edited before deployment. It's also a maintenance and support nightmare. Even better, it requires every user to maintain config patches and merge them whenever upstream code changes.

Currently, there is no standard deployment-time mechanism for overriding bundled settings in web.xml, [container]-resources.xml, [container]-web.xml, etc with deployment specifics like local group names, local database hosts and ports, etc. A mechanism like this would greatly improve the utility of concepts like resource-ref based indirection and principal-to-role mappings.

Containers should also allow web admin console based management of things like role mappings post-deployment.

Glassfish 3.0 was released prematurely, and has buggy CDI until 3.1.1

Glassfish 3.0 had a sufficient number of major CDI bugs that it was practically unusable with Java EE 6 using CDI. Even Glassfish 3.1 needs a manual update of weld-osgi-bundle.jar to weld 1.1.1 be usable with Seam 3 Solder, Seam 3 Faces, etc, the use of which is necessary to paper over some of the nastier holes in the EE 6 specs.

See my previous posts for lists of CDI bugs in Glassfish, etc.

While not a Java EE 6 spec issue as such, the quality problems with Glasfish early on were a major problem for me and I suspect they caused a significant number of interested people to simply drop the platform and move elsewhere. I persevered, kept reporting bugs, and kept on chasing beta releases, but I've wasted weeks of development time chasing container and platform bugs that I should've spent on my application code. Most people can't afford to do that.

Someone else in Java EE 6 pain

2011-07-13T13:03:00.006+08:00

I'm not alone!

Harald Wellmann writes about his experience moving a Tomcat-based app to Glassfish, JBoss AS 6, and Resin. Warts and all.

I think Java EE 6 really needs more real-world attitude and less hype. More bug fixing and more spec tidying, less focus on the next whizz-bang feature. It's nice to see somebody else who's had similarly painful experiences with it. Perhaps I'm not stupid after all, I simply make the foolish mistake of expecting released software like Glassfish 3.0.1 to have most features fairly solid.

Harald also notes that CDI was a huge problem with Java EE 6 until quite recently, largely because Glassfish 3.0 was half-baked and under-tested; JPA 2.0 is prone to exciting implementation differences and bugs, and more.

My favourite quote:

Legacy

Backward compatibility is a good thing for veteran users of a framework. Users don't want to change all of their application code just for upgrading to a new framework version.

On the other hand, backward compatibility can be extremely confusing to new users: There's two or three solutions for the same kind of problem, the legacy one is not marked as legacy or deprecated clearly enough, tutorials and example code still use the old style, and you can only resort to your favourite search engine or to trial and error to make things work consistently.

(Emphasis mine)

That's a perfect description of some of the problems I encountered when trying to get a handle on the released standard Java EE 6 and Glassfish 3.0, which I foolishly assumed was a stable platform to build my app on as I moved into the Java EE world

Java EE 7 really needs to modularize the backward compatibility crap into separate archives so that apps can depend on either javaee-7 or javaee-7-legacy as appropriate. That way we'll finally be free of the two copies of the obsolete-before-release @ManagedBean annotation and all the other horrible legacy crap.

Java EE application servers - learning from the past

2011-06-25T17:13:00.011+08:00

(Edit 15 July 2011: JBoss AS 7 is here, and brings a huge improvement to class loading and memory management. It's not full isolation, but it limits the exposed contact surface between app server and app greatly, and massively improves class loading. Brilliant!)

If you've used a Java EE application server like Glassfish or JBoss AS for long, you will have experienced classloader leaks, though you may not have realized it.

If you've ever seen the error java.lang.OutOfMemoryError: PermGen space at deploy-time, read the linked article above. If you have ever worked on the JVM, on app servers, or on EE applications, please read on.

Even if you haven't hit classloader leaks, you should be aware of the various ways Java EE applications can cause memory leaks in the server and what to do about them.

For those coming here for help fixing an immediate issue with their app: Read the links above, and this article on using jhat's JavaScript interface to find likely leaks. More fancy JavaScript OQL tricks are here.

Background: classloader leaks

Classloader leaks are one of several symptoms of a design flaw in all current application servers. The memory space of the server is not clearly separated from that of applications running on the server. The Java Security Manager (if enabled) prevents apps from casually reaching into the appserver's memory to patch and tweak things. Unfortunately, this doesn't help the application server free all the memory used by an app when undeploying (killing and uninstalling) an application.

Any class in the application server its self, the core Java libraries, or additional libraries deployed to the app server for use by all classes may cause a classloader leak. All it has to do is hold a regular reference (rather than a WeakReference) to an application-defined class in an object that isn't destroyed when the app is undeployed. The app defined class in turn holds a reference to the classloader that loaded it. That reference keeps the app's old classloader alive, preventing that memory from being freed when the app is undeployed.

Since the Java libraries were not designed with application servers in mind, they're full of places where exactly that can happen. The article linked to in the intro shows one such case in java.util.logging .

One could try to clean up all places in the Java libraries where references to user-defined code may be held, but it's going to be a losing battle. Worse, it won't help protect the app server from libraries like JDBC drivers that typically get deployed to the server for the use of all apps.

Even if you did that, you still wouldn't have solved the problem. Classloader leaks are not the only possible kind of leak, they're just one of the most common and troublesome. Classes and classloaders exist in the PermGen space in the JVM, which is generally of fairly restricted size. This tends to make the leaks more obvious because the app server starts throwing OutOfMemory exceptions when an app is redeployed.

Another big problem area is libraries that use ThreadLocal storage for caches. It seems like a great idea, as using a ThreadLocal means you don't have locking overheads and delays. Unfortunately, in an application server environment things like HTTP threads are pooled between all applications on the server. If you have 200 HTTP threads and one of your applications likes to keep a 5MB ThreadLocal cache, even if that app isn't used very often you'll eventually land up with 1GB of memory used by those caches!

My understanding is that it's hard for the application server to clean up such caches. It can't tell which application owns which ThreadLocal data so it doesn't know what to get rid of at undeploy time.

Lessons from the past

Rather than trying to manually patch up every leaked reference and fix every library that uses a ThreadLocal cache, why not learn from operating system design?

Old operating systems used to share the same memory space between the OS "kernel" such as it was and the application(s). Apps could freely reach into kernel space using a simple pointer. A mistake in pointer arithmetic or numerous other errors could cause the app to trample all over OS memory. Perhaps more importantly the app had to carefully release any resources it acquired from the OS, otherwise the OS wouldn't regain the use of them until it was rebooted.

Modern systems protect themselves from processes running on them by isolating each process in a separate memory segment. The process cannot access the kernel's memory or that of other processes directly. Access to other processes is done via the kernel, and access to the kernel is done via a very restricted interface (usually a trap). Not only does this have huge advantages for security and protection against accidental memory corruption, but it means the OS can free all the memory used by a process by dropping the memory segment(s) it allocated for that process. It doesn't have to care about individually cleaning up each little string, struct and class allocated by the process, and it doesn't care if the application has memory leaks because they're confined to the application's process space so the leaked memory is recovered along with everything else.

Applying the lessons to EE app servers

How does this apply to Java EE application servers? Two ways. First: Cleaning up guest apps in detail and trusting them to behave properly is a losing battle, you need ways to brutally sweep away the remanants when you terminate an app. Second: Modern OSes already provide the required tools by running JVMs in protected memort spaces.

Future Java application servers should run each application in its own JVM, using efficient IPC mechanisms for server/client communication. References to appserver-provided objects held by the client would be proxies that used the IPC mechanism to do their work. Because no IPC mechanism is perfectly efficient, this means that some of the application server code would probably be run as a library within the application's JVM. For example, CDI/Weld would need to run in the app's space, as would any JDBC drivers, Facelets libraries, JAX-RS, etc.

To keep things reasonably efficient, the app server kernel would need to be able to pass sockets (HTTP connections etc) to client JVMs to avoid the overhead of copying HTTP replies via the master instance, but that's possible in most if not all OSes.

By clearly defining where the application server core ends and the application begins, it'd be possible to finally fix classloader leaks and all the other issues of the shared process space once and for all.

I'm well aware that this would have performance consequences, and that lots of work would be required to restructure app servers to be able to run this way. It's highly likely that core JVM changes would be required too. Nonetheless, if JavaEE app servers are ever to become as dependable as the OSes they run on, it's going to have to happen.

Right now, the common workaround to the problem is to run each application in its own application server domain, running on its on JVM listening on its own ports, etc. When apps are re-deployed, the app server domain is restarted to free leaked classes. A front-end server is usually employed to redirect HTTP requests to the appropriate app server(s). This model is even less efficient than the one I describe above, because it prevents app server instances from sharing anything.

Database preferences and product selection methodolgy

2011-06-24T18:03:00.005+08:00

I recently stumbled across an interesting weblog post by a DBA who expresses a strong preference for Oracle over PostgreSQL. I thought I'd respond to it with a few thoughts here, not so much because of the opinion expressed as the reasons given for it.

There are certainly plenty of areas where Oracle is a better technical choice than Pg for the job, so someone preferring Oracle isn't overly surprising or controversial. What surprises me is that the author expressed such a blanket opinion without apparent concern for use-cases or needs, instead appearing to assert that Oracle is universally better and that Pg is a second-rate Oracle clone. The preference seems to be not so much for technical reasons or need-fit reasons as political/ideological ones.

Product selection, values, and selection methods

I think this is a worldview/culture difference as much as anything. If you or your management want "somebody to sue if it breaks" and a single organization to assign blame to, Pg isn't for you. If your management always wants to buy "the market leader" irrespective of other concerns, Pg isn't for you. If you want to use the same product for absolutely every conceivable project, no matter that consistency costs you, then Pg isn't for you. For the latter case Oracle might not be either.

On the other hand, if your organization is willing and able to evaluate its needs and requirements then select products based on those needs, you might find that Pg is a good fit for some (though perhaps not all) the roles or projects you need a DB for. This might mean that you need to support more than one database if you find that you have a job Pg isn't well suited to. There certainly are things Pg doesn't do well. On the other hand, it also means that you may be able to save eye-popping sums in license fees by using Pg in roles where it is a good fit.

For example: If you want a multi-master shared-storage cluster on a SAN, Pg is a poor fit for that role. Similarly, if you want really strong XML and XSLT support in-database, Pg will probably not be good for you. On the other hand, if you need a high-performance master plus some read-only replicas for a system handing geographic information, that's one of the many areas where Pg is an ideal fit. For tasks where reliability and good performance are important and a solid relational database is required, Pg is a strong contender. If you determine what you need you can make informed decisions about what product(s) fit those needs on a case-by-case basis.

Support

As for support - personally, I've had nothing but bad experiences with single-vendor support. I've had particularly horrible experiences with the "support" offered by Adobe, by Quark, and by a few more specialized newspaper-industry products used at my work. I'm reduced to begging them for a fix, which they will provide according to their determination of the problem's importance and on their schedule. Support contracts have - in my experience - surprisingly little effect on this. Not only can I not fix a bug/fault in most big single-vendor software myself, but the vendors never provide debugging symbol data, so I can't even hook up a debugger and get a backtrace to analyse a crash to see what's causing it so I can try to work around it locally. Microsoft is one of the few vendors who provide debug symbols, and even then not for all their products.

By contrast, when working with Pg I've usually seen bugs reported by me or others fixed in hours, not days. For crashes (and yes, crash bugs do turn up) I can send off a complete backtrace showing exactly what went wrong and where, simplifying support and bug fixing immensely. In one case (multiple keys in a JDBC keystore using client cert auth) I landed up fixing it myself and sending in a patch that was included in the next version. I would never have been able to do that without the sources and debug symbols.

Sometimes a fix will hit git master minutes after a mailing list post points an issue out, typically along with additional regression tests to catch it if it comes up again. A few key committers (you know who you are) are absolute heroes in this area. Of course, responses do depend on whether there's anyone interested in the problem, how busy everybody is, etc ... and if you're using Pg for free you have no right to demand anything more.

Because you may not want to be dependent on the attention span of mailing list readers, paid support is available from numerous outfits, including several run by major Pg core contributors. By signing up with them, you get to assert your priorities and do things on your timelines.

What that means is that you have solid support guarantees if you want them, but the freedom not to take them if you don't need them for a particular deployment role.

Pg as a poor Oracle clone

I'm afraid this is just misinformed. Pg *does* try to maintain a degree of Oracle compatibility for ease of porting and because users want it, but it is not and has never been an Oracle clone. Please see http://www.postgresql.org/about/history.

In conclusion

Personally, I think product ideology is dangerous, no matter which product is your idol. Select according to your needs.

Solo sysadmin/coder

2011-05-31T08:58:00.003+08:00

Are you a solo coder who doubles as a sysadmin or some other role(s) in your organization?

I share your pain - and I want to share these articles with you, because if you struggle with it too they will change how you look at it.

The Coding Horror article was written by Jeff Atwood, a damn fine developer and co-founder of Stack Overflow / Stack Exchange, which to me gives it a bit of added oompf.

While you're at it, read this article on programmer's bad habits ... because you probably do at least one of them. I know I do. They're counterproductive and just make you feel worse, so being aware of them helps.

Of course, nothing is as good as saying "I'm dropping all my other roles, you can hire someone else for them. I'm just programming now, that way I can actually finish the projects I started two years ago." I only have to hold out another two months before the new guy at work is ready to take over :-)

Using Seam 3 with Glassfish 3.1

2011-05-25T10:39:00.010+08:00

Seam 3 builds on top of the Java EE 6 standards, filling in holes and omissions in the functionality provided by the specs and providing important enhancements that many programmers would otherwise find themselves implementing themselves. Seam 3 Solder in particular addreses some of the frustrating limitations in the CDI and Java EE 6 specs, providing well-engineered and easily re-used solutions to common problems.

Being a JBoss project, Seam 3 is unsurprisingly better tested on the JBoss application server. However, its goals explicitly cover portability and it's supposed to work on Glassfish and - where possible - even on servlet containers like Tomcat and Jetty. Perhaps unsurprisingly, Glassfish doesn't get as much attention , so it's a bit harder to use Seam 3.0.0.Final on Glassfish than it is on JBoss. It appears that it was also harder for the Seam 3 folks to get quick fixes for the many Glassfish bugs they found into Glassfish than it was for them to get fixes into JBoss AS, so many of the fixes for issues found in Seam 3 won't hit a Glassfish stable release until 3.2 comes out, if then.

In the mean time, there are some quirks to work around. The Seam 3 project documents Glassfish 3.1 compatibility issues here and you should read that document before continuing.

Many of the issues are solved pretty easily once you know what to do and where to look. It's well worth using Seam 3 instead of rolling your own solutions to many of the problems it tackles, so read on.

Update Weld (the CDI implementation) in Glassfish

Some of the bugs were in Weld. Those have been fixed, but Glassfish has not yet been updated to the latest Weld release. You can, and should, update Glassfish to the latest Weld yourself by updating weld-osgi-bundle.jar according to these instructions on the Seam Glassfish 3.1 compatibility page.

This will resolve several issues immediately, and will get you side-benefits like improved error messages from Weld in some previously hard-to-debug cases.

Work around Glassfish's over-strict class scanner

The biggest issue you will run into is a Glassfish limitation (GLASSFISH-14808) that causes Glassfish to try to resolve a reference to a class that is supposed to be hidden by the @Veto annotation and to abort deployment when the resolution fails. Because of this problem, dependencies of Seam 3 modules that are supposed to be optional become mandatory on Glassfish 3.1.

There doesn't seem to be any listing of the dependencies of a given module when deployed under Glassfish 3.1, so it's a bit of a guessing game. One thing that will help you a lot, though, is the Seam 3 project's Maven infrastructure, which provides the Seam 3 BOM as a central place to manage dependencies rather than doing it directly in the parent POM. This wonderful choice allows 3rd party Maven projects to import the Seam 3 BOM into their dependencyManagement sections, making it trivial to use the correct versions of the correct optional dependencies and to use all the right versions of all the Seam 3 modules together.

To use the Seam 3 BOM, import it into your DependencyManagement section.

<project ...>
    ...
    <dependencyManagement>
        <dependencies>
             <dependency>
                <groupId>org.jboss.seam</groupId>
                <artifactId>seam-bom</artifactId>
                <version>${seam.version}</version>
                <scope>import</scope>
                <type>pom</type>
             </dependency>
             ...
        </dependencies>
    </dependencyManagement>
</project>

You may now specify Seam 3 dependencies without an explicit version, as if you'd declared them in your dependencyManagement section directly. For example, with Seam 3 Faces:

<dependency>
   <groupId>org.jboss.seam.faces</groupId>
   <artifactId>seam-faces-api</artifactId>
   <!-- Override the Seam 3 BOM managed version, because Seam 3 Faces 3.0.1 is required to use Seam 3 Faces on Glassfish 3.1 -->
   <version>3.0.1.Final</version>
</dependency>
<dependency>
   <groupId>org.jboss.seam.faces</groupId>
   <artifactId>seam-faces-impl</artifactId>
   <scope>runtime</scope>
   <!-- Override the Seam 3 BOM managed version, because Seam 3 Faces 3.0.1 is required to use Seam 3 Faces on Glassfish 3.1 -->
   <version>3.0.1.Final</version>
</dependency>
<!-- Required by Seam Faces (via Seam Solder and/or Seam International) on GF 3.1, but not explicitly depended on -->
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <scope>runtime</scope>
</dependency>
<!-- Required by Seam Faces on GF 3.1, but not explicitly depended on -->
<dependency>
    <groupId>com.ocpsoft</groupId>
    <artifactId>prettyfaces-jsf2</artifactId>
    <scope>runtime</scope>
</dependency>

Using the Seam 3 BOM won't magically add the optional dependencies that're required on Glassfish 3.1, but it'll make managing them easier. You can still override versions specified by the BOM, as shown above where we force the use of Seam 3 Faces 3.0.1 because Seam 3 Faces 3.0.0.Final didn't work quite right on Glassfish 3.1.

If you deploy a project that uses Seam 3 to Glassfish and deployment fails with classNotDef errors, most likely you need to add another optional dependency that turns out not to be optional on GF 3.1. Look up which artifact the missing class is in and add it. Make sure to add them as runtime dependencies so you don't accidentally introduce a real dependency on them in your code!

For example, if I deploy Seam 3 Faces without the above added dependencies I'll see errors like:

SEVERE: Class [ org/joda/time/DateTimeZone ] not found. Error while loading [ class org.jboss.seam.international.datetimezone.DefaultDateTimeZoneProducer ]
...
SEVERE: Exception while loading the app
SEVERE: Exception while loading the app : org/joda/time/DateTimeZone
java.lang.ClassNotFoundException: org.joda.time.DateTimeZone
 at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1518)
        ...
...
WARNING: The log message is null.

if JodaTime is not added as a dependency. Similarly:

Error occurred during deployment: Exception while loading the app : com/ocpsoft/pretty/faces/spi/ConfigurationProvider. Please see server.log for more details.

SEVERE: Exception while loading the app
WARNING: The log message is null.
SEVERE: Exception while loading the app : com/ocpsoft/pretty/faces/spi/ConfigurationProvider
java.lang.ClassNotFoundException: com.ocpsoft.pretty.faces.spi.ConfigurationProvider
 at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1518)
 at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1368)
        ...

if PrettyFaces isn't added.

Weird BeanManager exceptions

On deployment I also sometimes encounter an incredibly verbose 300+ line spew of exceptions including key lines:

CONFIGURATION FAILED! Failed to locate BeanManager using any of these providers: org.jboss.seam.solder.beanManager.DefaultJndiBeanManagerProvider(11), org.jboss.seam.solder.beanManager.ServletContainerJndiBeanManagerProvider(10). Please see server.log for more details.

org.jboss.seam.solder.beanManager.BeanManagerUnavailableException: Failed to locate BeanManager using any of these providers: org.jboss.seam.solder.beanManager.DefaultJndiBeanManagerProvider(11), org.jboss.seam.solder.beanManager.ServletContainerJndiBeanManagerProvider(10)
 at org.jboss.seam.solder.beanManager.BeanManagerLocator.getBeanManager(BeanManagerLocator.java:91)
 ...

SEVERE: PWC1306: Startup of context /myapproot failed due to previous errors
SEVERE: PWC1305: Exception during cleanup after start failed
org.apache.catalina.LifecycleException: PWC2769: Manager has not yet been started
 at org.apache.catalina.session.StandardManager.stop(StandardManager.java:872)
 at org.apache.catalina.core.StandardContext.stop(StandardContext.java:5509)
 ...

Stopping and re-starting Glassfish then re-deploying tends to resolve the issue, and it only happens after another failed deployment. It's not clear why it happens yet.

Scribd is amazing

2011-04-11T14:19:00.004+08:00

I've recently been looking for better ways to publish online editions of the newspaper I work for. We don't have a full content managed workflow internally, nor do we use automatic layout, so CMS-driven cross-publshing isn't currently an option. We're stuck with finding something to do with the PDFs that come out of the print production process for the forseeable future.

Scribd appears to be a much more impressive option than direct PDF publishing or tools like flexpaper & pdf2swf. The HTML5 viewer is amazing. Note how all the elements are selectable, columns have been correctly recognised in the pdf text, etc?

Here's an example.

This is the iframe-based embedding view, showing the HTML5 document viewer. There's also the option for native (iframe-free) embedding, but unfortunately it's Flash-only at the moment.

View fullscreen

POST Newspaper 2011-04-02 Small

Useful things I've discovered about Glassfish (especially embedded glassfish) lately

2011-03-31T15:31:00.011+08:00

I've been working with / struggling with Glassfish a lot lately. The documentation leaves much to be desired, so there are a few bits and pieces you might not know that might come in handy.

These notes refer to Glassfish 3.1.

Glassfish properties

Use asadmin list-commands to list supported commands, and asadmin help commandname for details. Read all of asadmin help since it contains important info the subcommand references should mention, but don't.

Use asadmin list * to list all Glassfish properties. This is important when working on embedded, because most configuration can only be done using asadmin set. In embedded glassfish you can run this on your embedded instance using:

commandRunner = glassfish.getService(CommandRunner.class);
CommandResult result = commandRunner.run("list","*");
System.err.println("Got command result: " + result.getOutput());

Use the get and set commands to manipulate the objects you discover with list.

Creating users in a file realm

Creation of users non-interactively - for batch/script use - is possible using the asadmin create-file-user command. It's not possible to pass a user password to the command directly because it rejects the --AS_ADMIN_USERPASSWORD argument with "passwords not allowed on command line". You must create an asadmin password file that contains the new user password to set as well as the admin password required to use the admin command. See asadmin help for details on the password file's weirdness and on the AS_ADMIN_USERPASSWORD variable. help create-file-user doesn't mention it at all.

maven-embedded-glassfish-plugin

The maven-embedded-glassfish-plugin is very limited. It doesn't offer facilities to deploy a glassfish-resources.xml, run asadmin commands pre/post deploy, etc. If you're using container provided resources, avoid it and run an embedded Glassfish instance directly in your application or tests instead. The Glassfish embedded API (see below) is pretty easy to use and unlike the Maven plugin gives you the flexibility to invoke asadmin commands to set up your domain to fit your needs.

There are references to a different maven plugin with a slightly different name out there. It's not the same thing, doesn't exist in Glassfish 3, and won't do what you want.

Documentation for the embedded glassfish 3.1 api

The Glassfish 3.1 embedded documentation is here: http://embedded-glassfish.java.net/nonav/apidocs/org/glassfish/embeddable/GlassFish.html. Most of the documentation has been removed from the Oracle Glassfish Embedded guide. The API has changed significantly from 3.0, so the 3.0 guide is just misleading.

`asadmin` from embedded

If you run your own standalone embedded glassfish, you can invoke asadmin commands using the CommandRunner. However, a frustrating bug caused by a feature removed without considering the impact on embedded means you can't create file users for a file realm this way. See http://java.net/jira/browse/GLASSFISH-16277. I don't have a solution for this issue yet.

REST admin

Glassfish has a REST driven admin interface in addition to asadmin and the web console. This can be a powerful facility for automation, but is almost totally undocumented. Exploring it using `curl' can be informative, because it's based on JAX-RS so paths can report the supported subpaths, etc.

JavaServer Faces (JSF2) and JAX-RS don't play well together

2011-03-11T14:54:00.017+08:00

I've been working with JavaServer Faces 2 (JSF2) since shortly after its release, and it continues to be an annoying mix of excellent and frustrating to work with. It's theoretically a great technology, especially with CDI (contexts and dependency injection) even despite the holes and quirks in the spec. There are just two wee problems.

First, unlike many of the other EE specs, the JSF2 spec really isn't written to be useful for developers intending to use JSF2 in their apps. It's very much a specification not a reference, and I found it hard to read even as a specification. Because I was getting started with JSF2 in the very early days of EE 6, I seem to remember that the now very helpful Java EE Tutorial wasn't EE6 ready yet or was incomplete, so I didn't have that as a basis for learning JSF2 and EE6 either. Perhaps my memory is just failing me. In any case, I often landed up stumbling through Google-land trying to find out details that I thought should be obvious and easy to find in the spec or other reference documentation. At the time, Google-land was full of outdated references to JSF 1, lots of migration guides from Spring to EE 6, and plenty of other things to side-track a confused newbie. All in all, it was really hard to get going with JSF2 alone.

Second, the JSF2 spec was clearly concocted with no thought of or co-operation with the JAX-RS spec, and it shows. They don't work well together, and it's really, really frustrating because they should be so useful in combination.

Edit: Part of the issue here appears to be that I was trying to use JSF2 just for the Facelets templating language, expecting to be able to interoperate with JAX-RS where possible. JSF2 is a framework and doesn't isn't really designed to enable you to easily step outside the abstractions imposed by the framework - not least because you shouldn't need to. I was probably going about this the wrong way in a square-peg, round-hole kind of way, and shouldn't have been trying to use JSF2 in the first place. That's not made particularly obvious by the docs, though.

Documentation

How do you insert the context path of your servlet into the page? The answer is to use the contextPath property of the implicit request property - but good luck finding that out unless you already know about it, or you've somehow managed to wade through the JSF 2.0 spec to find it. I was using FacesContext to obtain it via a very roundabout route before I finally found out about the implicit request object. There's plenty of information about the pageContext object from the older JSP standard, but that's defunct in JSF2.

Reading this might help you with JSF implicit variables. I wouldn't recommend trying to read the JSF2 spec its self as a reference and learning aid, though, it's really focused on spec implementers and isn't especially accessible to JSF2/facelets users. This one, at least.

JSF2 doesn't mesh well with JAX-RS

The other thing that's increasingly frustrating is the mismatch between JAX-RS (Jersey etc) and JSF2. They have different injection points in different lifecycles, making it hard to make a single class cleanly handle both JSF2 and JAX-RS requests. For example, if your method needs access to the HttpServletRequest, in JAX-RS you add a @Context HttpServletRequest request method parameter, which JAX-RS will automagically pass when calling your method. In JSF2 you access it via (HttpServletRequest)FacesContext.getInitialContext().getExternalContext().getRequest(). These two mechanisms are incompatible, and mean that you have to write two wrapper methods to call the same underlying code depending on how the call came in. Needless to say, that's annoying - and it's enshrined in the JAX-RS and JSF2 specs.

While you can call the JAX-RS method using EL method calls like ${myClass.myMethod(request)} you can't use that as a writable EL property. You land up writing SIX methods (JSF2 getter, JSF2 setter, JAX-RS getter, JAX-RS setter, private impl getter, private impl setter) instead of just annotating the two real implementation methods and letting injection take care of it.

Instead of something like the (non-working) code:

@Named
@Path("/test")
@RequestScoped
public class Something {

  @Inject private HttpServletRequest currentRequest;

  @Path("/property")
  @Produces("text/plain")
  public String getProperty() {
    // do the work here
  }

};

we land up writing something like:

@Named
@Path("/test")
@RequestScoped
public class Something {

  @Path("/property")
  @Produces("text/plain")
  public String getPropertyJAXRS(@Context HttpServletRequest request) {
    return getPropertyImpl(request);
  }

  public String getProperty() {
  getPropertyImpl((HttpServletRequest)FacesContext.getCurrentInstance().getExternalContext().getRequest());
  }

  private String getPropertyImpl(HttpServletRequest request) {
    // Do the work here
  }

};

This might be fixable with a CDI extension to provide consistent injection of things like the HttpServletRequest, but Seam Servlet (which should theoretically do it) fails to deploy to Glassfish with an NPE. Yay!

Of course, someone will chime up and say "you shouldn't need to do that, stay within the framework". Unfortunately web frameworks like JSF2 have large feature holes that sometimes force you to go to a lower level, especially if you need to inter-operate with other tools or frameworks. How do you get to the application's initial context parameters via facelets-only or jax-rs-only API, for example? You can't, you need to access the ServletContext to do that. How do you accept a multipart/mixed file upload with JSF2? Or exchange JSON data with a client?

Another problem is the difficulty of accessing the JSF2 context to do things like map outcomes to URLs from JAX-RS methods when you want to do something like send a redirect from a JAX-RS POST request to a JSF2 page.

Results

The end result is that whenever I need to add web services capabilities to my beans, I land up converting them to pure JAX-RS beans and converting my JSF2 xhtml across to plain XHTML+JavaScript with JQuery. So much pain just goes away when I'm not trying to mix both systems.

File upload webapp

2011-03-11T12:45:00.004+08:00

I recently became frustrated with the lack of easily self-hosted file upload app options. I needed to replace my employer's reliance on giant email attachments with something that wouldn't melt our mail server - and our clients could still understand how to use. This unfortunately ruled out the classic options like anonymous FTP.

In the end I wrote a new web application to accept files from clients over http and save them to a shared folder on the internal network. Many of the building blocks like uploadify were available, but there wasn't much around in terms of complete applications to do the job. Instead, people seemed to be using services like YouSendIt, Box.net, DropSend, etc ... which have downsides including:

Inability to provide a canned recipient list, or high per-user per-month fees to support one
Inability to customise the site's appearance and integrate it into your domain - without paying high fees for an "enterprise" version
The need to download files once the client has sent them to the provider, often one-by-one via a clumsy web interface
Windows-only client applications required to do batch downloads of files. Some services have Mac versions, but only one had a Linux client suitable for our advertising thin clients.
Lack of a web services API - unless you pay high fees for the "enterprise" version.
Lack of no-flash fallback upload methods
Sometimes iffy browser compatibility

I wanted something really simple for clients - and staff - to use, that simply dropped files into a shared folder as they were uploaded and sent a notification email to staff when the upload was finished. Now that I've finished it, I'm releasing the result as open source software for the use of anybody who wants it. You can get it here:

https://github.com/ringerc/postupload

It's a Java EE 6 project that runs on Glassfish. Don't stress if you've never used any of that before, if you only know PHP, etc. The client-side is all familiar HTML and JavaScript, and the server side isn't especially complicated code. Detailed instructions for installing and running Glassfish are provided.

Kobo and long-unfixed bugs

2011-02-26T12:00:00.018+08:00

(This post is part of a series on the internals of the Kobo and Kobo Wifi eReaders. The rest of the posts are technically focused; this one was just prompted by increasing frustration with the device.)

I really like the Kobo. It has great hardware, a great price, and some pretty nice software running on it. Unfortunately, it also has some annoying bugs and limitations - many of which wouldn't be too bad,if only there was any sign they'd ever get fixed/resolved. some but not all of which have now been fixed in the 1.9 firmware.

I feel a little bad paying the Kobo out about these issues, because the Kobo folks were really great to deal with and really responsive early on. I've been absolutely blown away by how far beyond narrow-interpretation GPL requirements they've gone with their source releases (including full build scripts and config files) and how quick they were to update their releases when asked about them. Please keep in mind that overall the Kobo is an excellent eReader and the Kobo guys deserve our support for being one of the depressingly few companies who fully accept, understand and act on the obligations that fall on them in exchange for the free use of GPL-licensed software.

(You can't) Go To Page (fixed in 1.9)

There's no way to jump to a page in a book by entering the page number, though the Kobo Wifi uses a handy on-screen keyboard for the online store that'd be ideal for the job. The inability to jump straight to page wouldn't be so bad, except that...

Reading position loss sucks

... it loses your page when it runs out of power. It tends to run out of power suddenly and without warning, which doesn't help. Indications are that `nickel', the reader app, doesn't checkpoint your reading position to the SQLite database periodically, it only does it on explicit shutdown or when you go to the home screen.

Factor in the fact that you can't jump straight to a page number, and you discover just how tedious it is to press "next page" enough times to get back to where you were. It's even worse in a book with long chapters, or with no chapter structure at all.

Which of the set I just added scrambled its brains?

How many files are really on this Kobo? Tip: more than "no books found".

(This issue updated 28 Feb to reflect new information; see comments)

At one point book I added to the unmodified orignal Kobo I had at that point caused it to crash during startup and reboot, failing to add that book to the database or recognise any new content added after that book was added^*. It rebooted normally into the usual `nickel' reading screen, but didn't see any of the new content. As I'd added a fair few books in that one update, I had no idea which book was the problem and no way to find out. I had to do a laborious binary search where I add/remove half the books at a time, slowly narrowing down the culprit. Tedious beyond belief, especially when it turns out you have more than one book that triggers a bug in Kobo's epub parsing.

To fix this, Nickel needs to write a checkpoint file indicating which book it's about to process, then remove it after successfully processing that book. If it starts up and discovers a checkpoint file in place, it knows it crashed while processing that book and should not try to process it again; instead it carries on with the next book. That way, the Kobo could tell you which books it failed to process, and let you read the rest of the ones you just added.

(Update: I recently had what I thought was another case of this, but even after a factory reset it still couldn't find any books, even previously working ones. A restore from a backup copy of my SD card fixed it. On balance, it's most likely that I broke something subtly ... somehow. The rest of the issues described here affect a stock Kobo, though, with no modifications.)

I'm not currently reading that (fixed in 1.9)

The Kobo keeps track of a "currently reading" list. This rapidly grows to include a long list of books you've looked at, decided you don't want to read right now, and left. You can't remove them from this list without clearing the SQLite database completely or deleting the book off the Kobo. A cosmetic issue, but it becomes an irritation to have many, many "1% read" books in your collection.

Dammit, which is the next book?

You know how series are very popular with authors and publishers these days? They bring in a lot more money, because people are willing to pay more for three (or five, or eight, or the godamm wheel of endless time ogod-make-it-stop-i-beg-you) books than just one huge one? And, frankly, they can be carted around in pieces rather than making you tote around something the size of the Necronomicon that could double as a potent self-defense weapon?

Well, there are lots of series out there. And the Kobo has absolutely no idea that they exist. Each book is individual and complete in and of its self to the Kobo, which doesn't seem to think that you REALLY want to read that six book series in order. I hope you have a smartphone to look up the Wikipedia entry on the series and find out which book is next, or you'll be doing plenty of guesswork.

Most ePub novels contain series and other metadata, but the Kobo doesn't use most of it.

Quote me once

The quotes are a cool idea. They're also hard-coded into Nickel, and there's only a small list of them. After a while, they may begin to drive you just a little bit batty. Why not put them in an XML file that's loaded at launch or on first use, so a bit more variety becomes possible? Or let us hide them? This doesn't bug me much, but it drives my girlfriend (who has a gen1 kobo) up the wall.

Don't just whine about it, do something!

I'd love to fix some of these. I'm a competent C++/Qt programmer, and I already have an ARM development environment set up for the Kobo that lets me compile binaries for the Kobo, including plugins for `nickel' (the Qt-based eReader app). Unfortunately, the KoboLabs GitHub repo doesn't contain any headers for Nickel and the plugin API its self is largely useless without them. Even with a usable plugin API, it's not clear that I could fix or improve many of these without the full sources to Nickel. So, really, all I can do is whine about it and hope the Kobo guys will get around to fixing some of these issues.

*Only once (the second time I thought it'd happened turned out to be a different problem) but it was very frustrating to track the issue down.

Drupal 7, PostgreSQL, unserialize, and bytea_output

2011-02-25T11:21:00.001+08:00

I just found out that PHP 5.3's PostgreSQL PDO driver (as used by Drupal) is broken by the Postgresql 8.0 transition from octal to hex encoding for bytea. It passes the raw hex through to the app, including the leading x , causing PHP's unserialize() function to choke.

I'm using Drupal with Apache and PostgreSQL on Windows (sigh) because I'm writing up a step-by-step hand-holding guide for someone who needs to do some testing against our Drupal database. I wouldn't be using that configuration voluntarily ;-)

Drupal doesn't check that bytea_output is set to 'escape' as a workaround or do a sanity test to detect this fault, so the results are ... interesting:

Notice: unserialize(): Error at offset 0 of 27 bytes in variable_initialize() (line 749 of C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\drupal-7.0\includes\bootstrap.inc).

If anybody else lands up whacking their head against this for a while: just

ALTER DATABASE drupal SET bytea_output = 'escape'

to have your sanity restored.

The amount of breakage being seen in drivers really makes me wonder if the the octal to hex transition should've been done with an opt-in from drivers during negotiation or an explicit SET, rather than just making it globally default. But, hey, hindsight.

Handling file uploads with Java EE 6 / JAX-RS / Glassfish / Uploadify

2011-02-04T14:03:00.009+08:00

Joeri Sykora wrote about using Jersey extensions to JAX-RS to handle file uploads in Java EE 6. His example is extremely handy, but needs some updates to handle Jersey 1.5.

Update: I've now put together a complete and self contained Java EE upload application using Uploadify, which you can grab from github.com/ringerc/postupload. See http://blog.ringerc.id.au/2011/03/file-upload-webapp.html

Jersey 1.5 and jersey-multipart's `@FormDataParam`

Jersey 1.5 advises that the jersey-multipart contrib module be used to handle uploads, using the multipart/form-data content type and @FormDataParam annotation.

Joeri's JAX-RS code may be adapted to read:

@Path("/file")
public class FileHandler {

  @POST
  @Path("/upload")
  @Consumes("multipart/form-data")
  @Produces("text/plain")
  public String uploadFile(
          @FormDataParam("file") InputStream file,
          @FormDataParam("file") FormDataContentDisposition fileInfo) {

    // your code here to copy file to destFile
    System.err.println("Received file: " +  fileInfo.getFileName() + " as " + file);

    return "1";
  }

}

... so that the file receipt is handled more efficiently.

Determining the correct path to the upload handler

Determining the URL to the upload handler isn't trivial. JAX-RS doesn't understand ../ paths, so faces/../rest/file/upload isn't the same as rest/file/upload to it. We can't easily create an absolute path, either, because the application is deployed in a configurable context root inside the Java EE server.

The easiest way to do it is build an absolute URL where you substitute the context root in. Assuming your JAX-RS servlet is configured to handle rest/* in web.xml and your upload handler is at file/upload in JAX-RS, you might use:

$(function() {
  $('#file_upload').uploadify({
    'script' : '#{facesContext.externalContext.request.contextPath}/rest/file/upload',
    // blah blah blah rest of uploadify config 
  });
});

Handling form data on a page that accepts Uploadify file uploads

An additional wrinkle is that Joeri's method isn't suitable for use where you want to have a <h:form> submit form data after all files are uploaded. If you attach the uploadify uploadifyUpload() call to a JSF commandLink or commandButton you'll find that the form submits before all the files upload. This is because uploadifyUpload() is asynchronous, returning before all files are sent.

To get your form submission to wait until all files are sent, you'll probably want to have your submit button (which should be an ordinary non-jsf button) call uploadifyUpload() ... then do nothing more. You can attach an onAllComplete() event handler to uploadify that invokes a hidden JSF2 commandLink when all the uploads are sent.

For example, add this to your uploadify ctor arguments:

'onAllComplete' : function(event,data) {
  // Escape due to css selector interpreation of colon
  // see http://docs.jquery.com/Frequently_Asked_Questions#How_do_I_select_an_element_by_an_ID_that_has_characters_used_in_CSS_notation.3F
  $('#upload\\:hiddenLink').trigger('click');
}

and use a jsf form something like this:

<h:form id="upload">
  <-- uploadify field -->
  <input id="file_upload" name="file_upload" type="file" />
  <-- Put your JSF2 input fields in here -->
  <input type="button" value="Clear Queue"  onclick="$('#file_upload').uploadifyClearQueue();"/>
  <input type="button" value="Submit Queue" onclick="$('#file_upload').uploadifyUpload();"/>
  <h:commandLink id="hiddenLink" style="display:none; visibility: hidden;"
    action="#{fileUpload.doUploadsSuccessfulAction}"
    value="doIt!"/>
</h:form>

... where String FileUpload.doUploadsSuccessfulAction() is a suitable managed bean that's ready to process the submitted form data.

How to get the context path (context root) in jsf2

2011-02-04T11:48:00.002+08:00

Use:

#{facesContext.externalContext.request.contextPath}

For some insane reason, none of the methods that should work would work when invoked via a @RequestScoped object with @Inject @FacesContext - they all merrily returned the empty string. More stupid CDI/JSF brokenness.

Setting up usb gadget serial (g_serial) on Kobo Wifi

2011-01-31T13:29:00.010+08:00

The Kobo Wifi (apparently) has an onboard serial port, but it doesn't have any header pins let alone a usable socket. Using it will require bulldog clips at best, more likely soldering some pins in. Either way you have to open the case and keep it open while using the port which is inconvenient if you like to actually use your Kobo. In any case, I can't find any documentation for the pinout.

Telnet over wifi works and can be enabled without opening up the device - but tends to go down quite a bit, as it's not really intended for this use. It's hopelessly unsuitable for running a GDB remote debugger over.

It's fairly simple to modify the Kobo Wifi (and presumably the original Kobo, though I haven't tried it) to add support for serial-over-usb and/or ethernet-over-usb, both of which are much easier to work with than physical serial and much better than telnet/ftp over wifi. Enabling USB serial gadget support seems to interfere with the USB gadget mass storage system used to export the file system to a host computer, though, so don't make the change permanent unless you like to keep your library on an SD card. It looks like the 2.6.33-rc1 and newer kernels may contain support for multiple gadgets running at once, but the Kobo is on 2.6.28 and it's unlikely to be worth the effort of an update.

To get usb serial mode working, you must build a new kernel for the Kobo and copy the modules from that kernel over to the Kobo. You might want to make menuconfig and enable CONFIG_USB_CDC_COMPOSITE in drivers -> usb support -> usb gadget -> CDC Composite Device if you'd like to have simultaneous support for Ethernet and serial gadget mode, which can be really handy for debugging. See documentation/README.kernel in the KoboLabs git repository for how to build a new kernel and modules. I've asked them to pull it, so it should be there soon. You should not actually need to install the new kernel, as the modules you build should be compatible with the old kernel already on the device.

To install the modules, install them to some temporary MOD_INSTALL_PATH like /tmp/arm then cd /tmp/arm and tar cvzf KoboRoot.tgz lib. Copy the KoboRoot.tgz file to .kobo/KoboRoot.tgz on the device's onboard user-accessible flash storage and reboot to install the update. After a reboot, you can telnet in and modprobe g_serial (or g_cdc if you compiled the mixed serial/ethernet module) to enable USB serial support. To make the serial port useful you need to run /sbin/getty -L ttyGS0 115200 vt100 to listen for logins. This is easily wrapped in a shell script you can invoke via telnet, or via an autorun hook you add in /etc/init.d/rcS that runs a script off an add-in SD card if one is found. The same script can be used to bring up the usb0 ethernet device if you're using g_cdc.

If you want to make usb serial gadget support start at every boot (thus disabling the ability to manage the internal card library over USB - this will be annoying!), add this line to /etc/inittab on the device:

ttyGS0::askfirst:/sbin/getty -L ttyGS0 115200 vt100

... and add this line to /etc/init.d/rcS after "/bin/mount -t sysfs" :

/sbin/modprobe g_serial

If you'd prefer to have serial and ethernet, and made the kernel config change to enable it as described above, replace g_serial with g_cdc in the line above.

Once you've loaded the g_serial module one way or another and started a getty, you can connect the Kobo to your PC. When prompted for what to do, say "keep reading". You'll discover that the Kobo appears as a USB serial port that you can connect a terminal to. On a Linux host it should be /dev/ttyACMx (/dev/ttyACM0 if you have no other USB serial ports); check dmesg to be sure. You can use a terminal program like minicom, gtkterm or picoterm to talk to the port. The login is "root" and there is no password unless you set one later. You can transfer files using zmodem's rx command if you need to, use the usb ethernet module, or ftp them over wifi.

Unlike telnet over wifi, this serial console stays up no matter what antics nickel performs, and doesn't require the device to be open. It's similarly easy to use g_ether to expose an Ethernet interface or g_cdc to do both ethernet and serial at once. It's just a real pain that using them prevents the g_file_storage module used by the Kobo to manage the onboard memory from working!

Compiling Qt plugins for Kobo Wifi

2011-01-28T20:21:00.004+08:00

The Kobo Wifi's main user interface application, nickel, is a Qt Embedded app. It drives the ePaper display via a Linux framebuffer interface and some ioctl calls to trigger display refreshes. This is abstracted away from the app behind a QWS (Qt Window System) driver called broadsheet_ioctl that's statically linked into nickel.

Because the QWS driver is only in nickel, we can't really write stand-alone Qt applications to run on the Kobo without reimplementing it. However, the nickel app has a simple plugin interface you can use to load your own code as a shared library.

To do this, you need a full Kobo development environment, including a cross-compiled Qt Embedded build that matches the one used on the Kobo. Once you have that, you can build apps according to the template provided by examples/poker, then drop the shared library produced into /usr/local/Kobo on the device to get it to load.

You now have some totally useless code running on your Kobo. Congratulations. There's no way to invoke it. For now, I'm testing by replacing the poker plugin with my own code, but it's still painful to access it so it's not desirable for real-world use. It may prove necessary to introspect the Qt widget tree or even runtime patch classes to get useful functionality loaded by plugins without having access to the nickel sources.

If your plugin doesn't load, it may be helpful to telnet into your Kobo and create a script like this, called (eg) /usr/local/relaunch-nickel.sh

#!/bin/sh
# Kill off the old nickel
pkill nickel
# Clear plugin cache
rm -f /mnt/onboard/.kobo/Trolltech.conf
# Restart nickel
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib:
export NICKEL_HOME=/mnt/onboard/.kobo
export LD_LIBRARY_PATH=/usr/local/Kobo
export QWS_KEYBOARD=netronix
export INTERFACE=eth0
export LANG=en_US.UTF-8
QT_DEBUG_PLUGINS=1 /usr/local/Kobo/nickel -qws -display broadsheet_ioctl >/mnt/onboard/.kobo/nickel.log 2>&1 &

When executed, this script will restart nickel, causing it to write a debug log to the .kobo directory on the user flash and print detailed information about plugin loading. This might help you figure out what's going on. nickel prints a fair few warnings during normal startup, so don't be thrown by that.

When developing for Qt on Kobo, the Qt for Embedded Linux Environment Variables document may be helpful, though it doesn't mention QT_DEBUG_PLUGINS.

It can also be helpful to determine how Qt was built on your target device. On the Kobo Wifi:

$ strings libQtCore.so | grep 'Build key'
Build key:           arm linux g++-4 full-config

Taking a disk image of the Kobo Wifi without opening the device

2011-01-27T18:30:00.007+08:00

Once you've enabled telnet on your Kobo Wifi, you can use it to transfer a disk image of the Kobo's entire firmware image on the internal MicroSD card. This will give you a backup copy of your Kobo's firmware on your computer.

You can't restore the firmware image without physically opening your Kobo if you mess things up badly enough for it not to boot, but hopefully you won't do that. The Kobo seems to be amazingly hard to break unless you corrupt its file systems, overwrite its kernel, etc. Anyway, this way you can avoid opening the Kobo until/unless you ever need a total firmware restore.

Imaging the Kobo's firmware over wifi

To image the disk of a running system, we need all its file systems in read only mode. This turns out to be trivial to achieve with the Kobo, though it requires terminating the main eReader process ("nickel") so you'll want to reboot after you've done it. During the process, the screen will display whatever was on it before you ran the commands to image the system and none of the keys will respond. Just:

pkill nickel
mount -o ro,remount /mnt/onboard
mount -o ro,remount /
nc -l 9984 < /dev/mmcblk0

Now, take note of the IP address printed by the commands above and use it in this command on your main computer:

nc IP_OF_KOBO 9984 > kobo_internal_microsd.img

The disk imaging process will take a while over wifi, and won't show any progress. Mine took about twenty minutes. If you want progress indication, open a second terminal and run:

watch "du -ms kobo_internal_microsd.img"

... to get a count in megabytes of data transferred so far.

Finally, reboot the kobo to bring it back to normal. You can do this by pressing and holding the power button for 6+ seconds, or by typing control-C then "reboot" and enter into the telnet command line on the kobo.

The image saved on your computer should be 1977614336 bytes for a Kobo Wifi with a 2GB internal card (~1.1GB user accessible memory).

If you have any issues, try a different port number with nc.

^C
reboot

Once you have the Kobo's disk image, you can mount the partitions within it to examine them, and you can extract some individual components of the unpartitioned space in the card.

Imaging the Kobo's firmware to an SD card

In principle you can also image the Kobo's firmware to an SD card instead of using wifi. This will be much easier if you don't run Linux. You need a 2GB or larger card.

To image the Kobo's firmware to an SD card you can telnet in to the kobo and make the file systems read-only as above. Make sure you unmount the external sd card /mnt/sd if it is mounted. Then, instead of getting the IP address and running netcat in listen mode, you can just `dd' the internal mmc card to the external sd card, `sync' to force it to flush, and `reboot'.

Alternately, you may prefer to leave the external SD card mounted, remount everything else read only, then run:

dd if=/dev/mmcblk0 | gzip > /mnt/sd/kobo.img.gz

... to make a compressed image (slow!) on the file system on the SD card. This will be easier to work with under Windows, which doesn't work well with raw devices.

Examining the firmware image

gunzip the image if it's compressed. Either back up a copy of the firmware image that you will not touch, or make the firmware image read-only with chattr +i.

You may now mount the three file systems on the image using this little script - or by hand by using "fdisk -l" to dump the partition table, then calculating the offsets to pass to losetup. If you want to use my script, I make no promises that it won't eat your system and your cat, so be careful. Save the following to "kobomount.sh" and mark it executable:

#!/bin/bash
set -e -u
if test $# -ne 1 ; then
  echo "Usage: $0 firmware.img"
  exit 1
fi
IMG=$1
i=0
sudo -v
for offset in $(sfdisk -d kobo_2gb_microsd_image_v174.img | grep start | cut -d : -f 2 | awk '{print $2}' | sed 's/,//g' |grep -v ^0$); do
  sudo losetup --offset $(( $offset * 512 )) /dev/loop$i "$IMG"
  sudo mkdir -p /mnt/kobo/$i
  sudo mount -o ro /dev/loop$i /mnt/kobo/$i
  ((i++))||true
done

You may then aim it at your firmware image and it'll mount the contained partitions on /mnt/kobo/0 (the recovery partition), /mnt/kobo/1 (the main OS) and /mnt/kobo/2 (the user flash partition). All will be read-only.

You may also want to extract some blobs from the unpartitioned space at the start of the image. That's where the kernel, boot splash image, etc live. I don't have official documentation on the layout of this space, but reading the upgrade scripts in /etc/init.d suggests that these values should be about right. They are UNVERIFIED except for the boot splash image, which I've been able to edit and replace.

IMG=kobo_2gb_microsd_image_v174.img
dd if=$IMG of=serialnumber.bin bs=512 count=1 skip=1
dd if=$IMG of=hwconfig.bin bs=512 count=1 skip=1024 count=2
# The image that ships with the kobo is < 470 512 byte blocks long. It's not clear
# if something else takes the space between the end of the image and the start of
# the epson display binaries, if it's dead space, or if there's room for bigger
# images.
dd if=$IMG of=bootlogo.bmp bs=512 skip=1026 count=470 
dd if=$IMG of=epson_display_setup.bin bs=512 count=1 skip=1920 count=8
dd if=$IMG of=epson_waveform.bin bs=512 count=1 skip=1928 count=$((2048 - 1928))
dd if=$IMG of=kernel.bin bs=512 count=1 skip=2048 count=$((7564-2048))

The Kobo's kernel

We can now examine the existing kernel to learn more about it. This post tells us how to extract the kernel (but it's more easily done with scripts/extract-ikconfig from a kernel tree)

The hard way (which lets us extract the image, not just the config) in brief:

od -A d -t x1 kernel.bin | grep '1f 8b 08 00'
# Note the (decimal) offset, correct for offset of deflate header into line
dd if=kernel.bin bs=1 skip=0013076 | zcat > vmlinux

The easy way to dump the config:

scripts/extract-ikconfig $HOME/kobo/kobofirmwaredump/kernel.img

Boot loading is apparently performed by RedBoot (2.0 according to shipped sources), which should offer tools like `fis list' to report on the structure of the redboot-managed areas of flash storage. Building redboot is a nightmare, so I haven't investigated this yet.

Enabling telnet and ftp access to the Kobo Wifi

2011-01-27T17:10:00.011+08:00

The Kobo Wifi has this handy 802.11 radio that it really only uses for online shopping. Not only is it calling out to be used for RSS feeds and web browsing, but it is also a really handy tool for developing on the device without having to crack it and solder serial port headers onto the board.

We can gain telnet and ftp access to the device quite trivially; see the pre-made KoboRoot.tgz patch linked to below for the really easy way.

If you want to roll your own patch or you're using a different firmware revision, there isn't much to do. The Kobo already has busybox's telnetd, ftpd and inetd on it. All that's necessary is to:

Edit /etc/inittab and add the lines:

::sysinit:/etc/init.d/rcS2
::respawn:/usr/sbin/inetd -f /etc/inetd.conf.en

Create /etc/init.d/rcS2 with the content:
```
#!/bin/sh
mkdir -p /dev/pts
mount -t devpts devpts /dev/pts
/usr/sbin/inetd /etc/inetd.conf
```
and run chmod a+x /etc/init.d/rcS2 to flag it executable.

create /etc/inetd.conf with the content:

# service_name sock_type proto flags user server_path args
21 stream  tcp     nowait  root    /bin/busybox ftpd -w -S  /
23 stream tcp nowait root /bin/busybox telnetd -i

tar our new /etc into KoboRoot.tgz

(Edited for safer method suggested by tjm)

KoboRoot.tgz may then be used to update the device's operating system by putting it in the .kobo directory on the user-accessible flash.

Because it's not designed to be accessible from the outside world, the Kobo doesn't have any root password set. Telnet and FTP access as root will be offered with a blank password by default. You can change that once you telnet in if you like, by typing "passwd" at the root prompt. This won't affect the device's normal operations, only telnet/ftp access.

I created a canned KoboRoot.tgz (for firmware 1.7 ONLY) with these changes to save you the hassle of making them yourself. It was made from the initscripts in firmware release 1.7.4; if you use it with any other firmware and it fails to boot you'll have to do a factory reset. Because there are no binaries in the patch, only a modified /etc/init.d/rcS and a new /etc/inetd.conf, you can easily verify that the code isn't malicious. It has two optional features you can uncomment in /etc/init.d/rcS to (a) syslog to user flash for debugging, and (b) run a user-defined script from .kobo/rc.sh every boot. Both are commented out so they do nothing unless you edit the script to enable them.

Unlike modifying the boot splash screen, this is a pretty safe change as it's completely erased by a factory reset of the device.

Changing the Kobo's boot image

2011-01-27T14:41:00.009+08:00

The Kobo's boot splash image is stored in unpartitioned space on the internal MicroSD flash. It's an 600x800 4-bit greyscale Windows bitmap (not RLE compressed) starting 1026 512-byte sectors from the beginning of /dev/mmcblk0 (the onboard SD).

Before trying to modify the boot splash image or anything else on the onboard SD card, make a backup of your Kobo's firmware by disk-imaging the internal SD card. If you don't back it up and you damage the kernel or bootloader you won't even be able to factory reset your device; it'll be useless.

If you use a bitmap that is too large you will render your kobo useless, so be careful. I've verified a bitmap of 240120 bytes, saved from Adobe Photoshop CS2, to work on my Kobo Wifi.

`file logo.bmp' should report PC bitmap, Windows 3.x format, 600 x 800 x 4

I haven't found any unix/linux based tools that will write 4-bit greyscale Windows Bitmap images yet, so Photoshop is your best bet. Recommendations appreciated.

The boot splash image may be replaced by putting a compatible bitmap on the user-accessible onboard fat32 partition. It must be located at .kobo/upgrade/logo.bmp. Once placed, disconnect the kobo. It'll detect the upgrade files automatically, apply them, and reboot. On reboot, it'll show the new splash.

(Sorry for the horrible phone-camera pic)

Preparing a development environment for the Kobo Wifi

2011-01-25T10:15:00.005+08:00

First, I'd like to thank the KoboLabs members for being so helpful and responsive as I try to get up and running with Kobo development. They've pushed several tools, config files, scripts etc to the KoboLabs repository, all of which have drastically reduced the amount of time it's taken to get up and running and largely eliminated the need for reverse-engineering work. This is incredibly unusual for an embedded company, and absolutely wonderful. I was already impressed by their ongoing releases of new firmwares for the original Kobo after the release of the Wifi, as most companies just drop supports old products. Now I'm blown away. Please tell people how damn impressive the Kobo folks are with their after-sales product support and maintenance; I know I will be.

In fact, this whole document has been rendered largely unnecessary by improvements to the KoboLabs code drops. They've posted their build scripts and configurations. To get started, just clone the git repo and read documentation/README.

Overview

You will need to download the CodeSourcery toolchain used for Kobo development, then build the libraries the Kobo uses so that you can link your new executables against them. Qt Embedded, with its dependencies, is particularly crucial. The latest GitHub code drop includes the Qt Embedded configuration used on the Kobo, letting you compile a compatible version of Qt.

Installing Sourcery G++

Download Sourcery G++ 2010q1-202 4.4.1 for your platform and run the installer.

If you're on an x64 RPM-based Linux distro, you may need to install a 32-bit userspace to be able to run the tools. x64 Debian/Ubuntu users will have to install them in the 32-bit chroot, and do all future steps described here within that chroot.

cd ~/Downloads
chmod a+x arm-2010q1-202-arm-none-linux-gnueabi.bin
./arm-2010q1-202-arm-none-linux-gnueabi.bin

Accept the default install path of $HOME/CodeSourcery/Sourcery_G++_Lite when prompted. Accept all other installer defaults.

You now need to symlink your tools to new names, as some of the Kobo makefiles etc expect the tools to be named arm-linux-x where they're named arm-linux-gnuegabi-none-x by CodeSourcery. To do this:

cd ~/CodeSourcery/Sourcery_G++_Lite/bin
for f in arm-none-linux-gnueabi-* ; do ln -s $f arm-linux-${f:23}; done

If all is configured properly, after a logout & login or after running "bash -l" you should now be able to type "arm-linux-g++" and get the response "arm-linux-c++: no input files".

Compiling libraries

Now that we have a toolchain in place, including gcc, binutils, glibc, and all of that provided for us by the wonderful folks at CodeSourcery, we can get on with the real work of preparing our dev environment.

Clone a copy of the KoboLabs git repository to get the required sources and patches:

mkdir ~/kobo
cd ~/kobo
git clone git://github.com/kobolabs/Kobo-Reader.git koboreader

This will take a while, as it's downloading the sources to a lot of libraries the Linux kernel used on the Kobo, and more.

Once you've cloned the repository, it should be easy to get going. A reasonable option might be:

echo "DEVICEROOT=$HOME/kobo/fs" > ~/kobo/build/build-config-user.sh
mkdir ~/kobo/tmp
cd ~/kobo/tmp
../koboreader/build/build-all.sh

... which will use the Kobo build scripts to compile all the libraries and install them in DEVICEROOT, which you set to $HOME/kobo/fs . The unpacked sources and temporary object files will be stored in ~/kobo/tmp , so your git clone of the kobo sources doesn't get messed up by the build.

Now you need to build Qt, which you can do by following the instructions in documentation/README.trolltech

Running your own code on the Kobo

2011-01-23T23:08:00.006+08:00

The 1.7.4 update tarball (see http://soapyfrogs.blogspot.com/2011/01/getting-kobo-update-urls.html) contains a new copy of /etc/init.d/rcS. Reading it tells us how to replace files on the Kobo's root file system, giving us a way to replace rcS with a patched version that calls out to a shell script on the more easily writable FAT32 main flash, so we can tinker with the system with minimal risk. This initial change is risky, though, as you might render your Kobo useless with a typo!

Replacing files on the Kobo Wifi's root file system

The Kobo Wifi's /etc/init.d/rcS startup script tests for the presence of a number of files within the .kobo subdirectory of the main fat32 partition.

Two notable files are .kobo/Kobo.tgz and .kobo/KoboRoot.tgz. If these files exist, they will be extracted into /usr/local/kobo or / (the root directory) respectively. The system uses this mechanism to apply incremental updates.

We can exploit this mechanism to insert our own files onto the device's main memory or very carefully replace existing files. If you screw it up you might render your Kobo useless and possibly irreparable, so don't try anything unless you don't mind breaking it.

You're much better off copying the firmware from the internal microSD card to a 2G SD card then booting off the external SD by holding "enter" down during power-on. Getting the image for this requires physically opening the Kobo to get to it, unless someone finds an image of it hosted by Kobo somewhere, though.

BTW, the rcS script also looks for .kobo/upgrade and if found, runs /etc/init.d/upgrade-wifi.sh . This is probably the hook for a full upgrade installation. Don't mess with it.

Getting Kobo update URLs

2011-01-23T23:00:00.004+08:00

Whenever the Kobo eReader connects to wifi, it sends a HTTP/XML request to the Kobo severs with some details about the device, including the current firmware version. The server sends a HTTP/XML response containing various URLs for the Kobo store, etc.

If the firmware version sent by the Kobo isn't the same as the current Kobo firmware, the server also includes the URL for a software update in the HTTP/XML post.

This is apparently only an incremental patch on the main firmware, not a full firmware download, so I still need a full firmware image and will have to extract the MicroSD card from the hardware to get it.

The 1.7.4 update

The URL for the 1.7.4 update is: http://download.kobobooks.com/firmwa...rade-1.7.4.zip

Examination of it tells us some useful things that help open the hardware up some more. See the next post.

How to get the current firmware URL from the server

You can find out the update URL by capturing the wifi traffic of a Kobo with an older firmware when it connects to the network. Use wireshark with the capturing machine's wifi card in monitor mode, or capture the traffic at the router. If you capture off wifi, you'll need to enable 802.11 decryption in Wireshark's protocols->802.11 preferences and enter the key of your network.

In your network capture, look for a request like:

POST /MobileRequest.ashx HTTP/1.1

and in the HTTP/XML response to it you'll find something like:

<UpdateURL>http://download.kobobooks.com/firmwares/kobo2/kobo2-upgrade-1.7.4.zip</UpdateURL>

If you don't have a Kobo with an old firmware, you can achieve the same effect by recording the POST request from an up-to-date Kobo, saving it, modifying it so that <ApplicationVersion>1.7.4</ApplicationVersion> instead reads (eg) <ApplicationVersion>1.6.0</ApplicationVersion>. Because changing the length of the request will corrupt it, you're best off using `sed' to do it. Eg:

sed -e '/^Accept-Encoding/d' -e 's/1.7.4/1.6.0/g' request.txt > request-mod.txt
nc mobile.kobobooks.com < request-mod2.txt > response-mod2.txt

The command deletes the request header that asks the server to gzip the response (for easier reading) and changes the version the "Kobo" announces its self to be running when it connects to the server.

(post series)

Kobo and Kobo Wifi exploration and enhancement

2011-01-23T12:03:00.020+08:00

This article is part of an extended series on Kobo development and investigation

I just got myself a new Kobo Wifi, after spending a year drooling over my girlfriend's Kobo 1st gen. I immediately felt the impulse to fiddle3 come over me, resulting in this collection of information about the hardware, firmware, etc as I research what I can do with it. The result is this series of posts.

There's lots to be learned about it from the sticky threads in the Kobo section of the MobileRead forums. I also strongly recommend reading their wiki article on the Kobo, including the FAQ.

About the Kobo

The Kobo is a great device - it has a few flaws, but may good points that greatly outweigh those flaws. The original Kobo was based on the Netronix EB600, the same board as the Cool-er reader, Astak Mentor Lite, booq, and numerous other devices. Netronix's website is scary-bad, but their products are pretty nice. It's not 100% clear if the Kobo Wifi is still an EB600 product - the kernel refers to the "imx357" - but it's certainly Netronix.

Kobo good points include price, format support, gorgeous eInk screen, light weight and nice feel, and a nice-to-use UI. It's also extremely moddable. It uses a nice standard USB mass storage interface for transferring books so you don't need dedicated vendor crapware applications. With the -wifi you can buy books directly on the device, and with the old version you can use the Adobe Digital Editions app if you don't want to use the Kobo application to buy DRM'd books. Support for DRM-free books is perfect, and is just a matter of dropping them onto the flash.

Bad points include somewhat slow menu navigation (some say page turns too, but I find those just fine), lack of full-justified text display option, useless Bluetooth in the 1st gen, and the inability to remove books from the "currently reading" list. It also has a long pause after unplugging it when you've added books, during which it scans them and adds them to the internal SQLite catalog. It's a pity the Kobo Desktop application can't be used for this while it's connected, though it's wonderful that the device can do it its self and doesn't rely on such an application.

Later firmwares improve some prior issues dramatically, allowing things like charging while reading, and are strongly recommended. The latest firmware is out for both the Kobo and Kobo Wifi; they haven't dropped support for the old generation of hardware, which they deserve real credit for. The use of FAT32 means it risks file system corruption if it's unplugged while transferring books - but this will only happen if you mistreat it, won't cause the device to fail to boot or anything, and is trivial to fix.

The Kobo is massively improved by the use of the free and open source Calibre e-Book management software, which is vastly superior to the Kobo Desktop application in every way. You don't need any management app at all (thanks to USB mass storage support) but Calibre is wonderful for document conversion and cleanup, management, automatic conversion of newsfeeds to ebooks, etc.

Kobo modding - hardware/software info

First: Do not try to reflash your Kobo with firmwares from other EB600 devices. Reports from the mobileread.com forums suggest that this will render your device nonfunctional and make it impossible to restore the original firmwares, though others report having successfully converted a Kobo into a Cool-ER Reader. If you try it, make sure you can afford to replace your eReader or are a JTAG wizard before you try converting it to the Cool-ER firmware or anything like that. At the very least, make sure you take a disk image of its MicroSD card first.

There is much to be learned about the Kobo and many other devices on the MobileRead forums. I'm ringerc there, though I'm not a big forum user so you're better off commenting here or emailing me.

The kobo's kernel, operating system, and user data is stored on an onboard 2GB MicroSD card. There's also a full size externally accessible SD card slot for add-on storage. There are three partitions on the internal microSD - a recovery root FS (ext3), the main root FS (ext3) and the user data partition (fat32). The fat32 partition is exported over USB as a mass storage device when the device is connected in "manage library" mode. The MicroSD card has some unpartitioned space before the recovery partition, where it stores the boot splash screen (800x600x4 Windows BMP), kernel, bootloader, hardware config data, epson display setup data, and probably more.

Because everything is on MicroSD, it's wonderful to hack on. Just pop it open, disk image the MicroSD card using a MicroSD adapter in a PC, and you can recover from pretty much whatever screwup you might make without attacking a flash chip with a soldering iron or learning your JTAG voodoo.

The Kobo (original) apparently presents a root console via its internal 3.3V RS232-like serial port headers, though examination of the Kobo Wifi's board suggests that it may lack these headers. This should allow full access if you're willing to crack your Kobo case - possibly literally, because it's made for easy assembly not easy opening, and is by all accounts a pain to open. I had no real problems opening my Kobo Wifi carefully using a butter knife to pop open all the clips around the edge of the front bezel, but I don't recommend that you try this unless you don't mind the risk of breaking it.

I've been informed by someone working on the Kobo that there is no "trusted boot" with signed kernel and bootloader. The Kobo uses u-boot (the Kobo Wifi may use Redboot; not sure yet) and a regular Linux kernel (2.6.28 for the -wifi). Kernel and u-boot sources, along with some userspace helper apps and other things of interest can be found in the Kobo git repository on GitHub. It's a binary drop of tarballs in git (ugh); there's no version history and no way to differentiate Kobo mods from the original Netronix-supplied kernel etc, but at least it's there. Props to gtalusan for posting the updated sources for the new Kobo Wifi model less than a day after I asked him about them. He's also been wonderful about posting updates, fixes, config files etc as I get further into setting up a Kobo development environment.

The numerous ways of updating the Kobo firmware and root file system

The Kobo has an amazing number of different ways to perform firmware updates, plus several ways to apply small incremental patches to the root file system contents.

Smaller firmware patches are applied by putting a file called KoboRoot.tgz in the .kobo directory of the main onboard FAT32 flash. Upon reboot, this is untarred into the / directory then deleted. Another file named Kobo.tgz (also in .kobo) is untarred into /usr/local/Kobo if found, making for somewhat safer updates when only the Kobo application is being changed. There also seems to be a mechanism to update the Kobo firmware using the .kobo/update file, but I recommend staying away from that.

Firmware updates to the Kobo may be performed by copying a new firmware archive file to a separate piece of FAT32-formatted flash. The device then uses that file to rewrite its firmware. There's no direct flashing over USB to worry about. The Kobo Wifi can download new firmwares over wifi instead of having them dropped on it via USB mass storage. Both the Kobo and Kobo Wifi can also be updated via the Kobo Desktop application or by a simple "cp" of the firmware file by putting the device in firmware upload mode then connecting it over USB.

USB firmware upload mode (which seems to use the minimalist linux recovery partition on the microUSB) is entered by turning the device off and unplugging it from USB, then holding down the Menu key then pressing power. Continue to hold Menu until "Initializing USB partition" appears on the screen. The device will show "Please connect your eReader to USB" on the display. Doing so will mount an empty 200MB FAT32 volume labeled "KoboUpgrade" from a USB mass storage volume from "Netchip Technology" (usb 0525:a4a5). You may safely unmount and unplug the Kobo without copying a firmware file to the partition; it will briefly report that it's "updating software" then reboot back into the old firmware with all your books etc unchanged.

Kobo firmware updates can also be performed via the SD card. Some Canadian users received SD cards with firmware updates for the original Kobo. Reports suggest that it prefers <= 2GB SD cards for updates (not SDHC). Instructions here. I have not tested this. Do not try to install firmwares for other eReaders on your Kobo unless you don't mind rendering it permanently non-functional.

The Kobo can boot off an SD card - just hold down the middle button of the D-PAD ("enter") during power-on, until the first five black boxes of the boot progress bar have filled in. This can be used to test new firmwares without rewriting the onboard flash.

I've yet to build a new firmware for the Kobo, though I've been using the KoboRoot.tgz patch method on mine in an attempt to get telnet and FTP access over wifi. Figuring out how it's put together isn't too hard, but it's unlikely I'll be able to get the rights to distribute new firmwares, so using the Kobo's firmware patch mechanism is likely to be safer and more useful.

The Kobo Wifi runs on an arm6 CPU on a Motorola MX35 3-Stack compatible board. It has 128MB of RAM.

[root@(none) /]# cat /proc/cpuinfo 
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 530.84
Features        : swp half thumb fastmult vfp edsp java 
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3

Hardware        : Freescale MX35 3-Stack Board
Revision        : 35120
Serial          : 0000000000000000

What runs on the Kobo

The Kobo Wifi runs Linux 2.6.28, glibc 2.11.1, Busybox v1.17.1, and Qt Embedded 2.6.2 among other things. A fuller list can be found in the KoboLabs git repository.

The busybox configuration is fairly complete, though it lacks a bbconfig command. The full list is easily gained by running "busybox" without arguments on the Kobo, so I won't reproduce it here.

The sd8xxx wireless driver is loaded as a module. Everything else seems to be built in to the kernel, but module support is present.

The Kobo Wifi binaries were developed with CodeSourcery Sourcery G++ Lite 2010q1-202 4.4.1. To build plugins for the Kobo Qt application, rebuild Kobo binaries, or build your own, you will need these tools.

PostgreSQL automatic crash dumps for windows

2011-01-02T11:00:00.006+08:00

It seems my first real* PostgreSQL patch has been accepted and committed! By the time patch review was done I think it was almost more Magnus Hagander's patch than mine, but I'm still very happy to have put it together and seen it through into Pg mainline. Thanks very much to Marcus for testing, enhancing and committing the patch.

PostgreSQL 9.1 will now have automatic crash dump generation under Windows. That'll allow Windows users to run their production sites with crash dumps so we can do post-mortem debugging without holding up their server or requiring them to run it under a debugger. That gives us a much better chance of tracking down the cause of hard-to reproduce or intermittent faults. To activate the crash dump collection feature, just create a crashdumps directory in the data directory and grant the postgres user (or whatever user your server runs under) "full control" of that folder in Properties->Security. No configuration file changes or server re-starts are needed.

Helpfully, this change will also let the majority who don't know what Visual Studio or windbg.exe are send crash dumps off to someone more experienced with the tools already configured. There should be much less need for this page - and while it took me long enough to write, I'll be glad to see it rendered unnecessary.

* The one-liner fix for the X.509 client certificate validation bug doesn't really count, despite the truly epic amount of convincing required to get the patch applied.

Kogan KGN1080P32VAA - Avoid this TV and any other with a CultraView CV119MA mainboard

2010-11-01T08:57:00.011+08:00

The Kogan KGN1080P32VAA TV is a lemon. Avoid it, and Kogan.

Support

Before I go on, I want to note that Kogan's support were responsive, if not helpful, about these issues. They acknowledged the EDID defect, and while they won't fix it or anything else that's wrong with the TV they did credit me $40 for my trouble, showing some sign they accept their error. Kogan have been much better than, say, Belkin when it comes to supporting their products. Given that they still haven't actually fixed anything, that says scary things about the industry, doesn't it? Anyway, here's why you should avoid this product, anything else by CultraView, and possibly anything else by Kogan:

720p EDID on a 1080p panel, mainboard programmed wrong

When I bought the KGN1080P32VAA I expected (and got) a 1080p TV with cheap and nasty LCD panel; this is, after all, a $600 1080p TV. Fine, you get what you pay for, and it's still a 1080p full HD panel.

What I didn't expect was an incorrectly programmed mainboard that sends the EDID for a 720p TV rather than a 1080p TV, making the TV not "full hd" at all (and in fact worse than native 720p due to scaling) for anything that respects the EDID! What does this say about quality control and product testing? I certainly didn't expect Kogan to have no interest in issuing a firmware update to fix it, given that the TV's mainboard, by physical examination determined to be the CultraView CV119MA is still in production and supports firmware updates via USB key. CultraView don't respond to queries and don't offer their tools or firmware for download without a username and password to their vendor/partner FTP site. Kogan say:

We have tried to get a firmware upgrade for you TV however we cannot get solution for this issue.

I would still suggest you to use the VGA as you get Full HD resolution on VGA without any issue.

Alas, there's only one VGA input and associated 3.5mm audio port. I didn't buy a TV with only one (working) input. Additionally, at 1920x1080 the (analog) VGA looks ... kind of nasty. This is partly explained by the picture issue described later.

HDMI audio and overscan

The problems with HDMI don't end with the EDID, though. The CultraView CV119MA, as shipped by Kogan, turns on overscan on digital inputs when it detects a HDMI audio signal. No HDMI audio, no overscan. HDMI audio, overscan. This means you can't get decent picture quality and HDMI audio at the same time - you have to endure the image butchery of overscan and overscan compensation, or have no audio on that input. Overscan makes no sense for digital inputs, where a perfect 1:1 pixel representation is sent and there are no scan line fringes, etc, to deal with. Sadly overscan does need to be an option because some older/broken digital output devices produce images with overscan for compatibility with ancient TVs, but should never be the default, and is something you must be able to disable. It certainly shouldn't be controlled by whether or not HDMI audio is being sent!

It's possible to disable overscan by disabling HDMI audio - if your device supports this. For my media PC, that requires telling the nVidia driver to completely ignore the EDID so it doesn't detect the HDMI audio capability of the output. I had to override the EDID anyway because the TV sends a totally bogus EDID with only 1080i and 720p resolutions over HDMI. There are a couple of ways to override the EDID though - I found that overriding just the resolutions left me with an overscanned 1080p image, while overriding the whole EDID got me a 1:1 pixel sharp-ish 1080p image. The difference: HDMI audio.

The TV doesn't offer a "1:1 pixel" mode in its zoom/aspect list, nor in its menus. The only way to control overscan is via HDMI audio. I'm gobsmacked.

Vendors unwilling to fix anything

Neither Kogan nor CultraView will supply a firmware update, or the tools to create one using the vendor customisation tools CultraView offer to vendors. Neither will supply the documentation for the mainboard, which has "PC DEBUG" (JTAG?) and "PC INPUT" (DB9 serial headers? Maybe?) on it and has updatable firmware. Examination of a firmware image I found on the 'net confirms that it's capable of being reflashed via USB.

Brightness controls pixel values not backlight

The problems don't end with a misprogrammed EDID and idiotic overscanning behaviour there, though. When you change the "brightness" setting in the TV, it doesn't adjust the backlight, it scales the pixel values down toward the black end, so you lose contrast and get horrid banding. While this can be written off as a "cheap TV" problem, it's one I never even imagined might be possible.

Horizontal blur at native 1080p

Worse: Even now that I've finally achieved a non-overscanned image at the panel's native 1080p resolution, the image is still crap. By creating a few stippled and striped b&w test patterns, I was able to determine that horizontal lines are sharp and clear - so there's no vertical scaling or distortion. An array of 1-pixel wide b&w vertical lines, though, are blurred and smudged when displayed on the TV, as if they've been scaled down then up, or up then down, before display.

The test image, unscaled, appears below. If your LCD display is set to native resolution and your browser isn't scaling the image, this image should appear as a block of fine vertical lines.

... and here's a photo of what you see on the Kogan TV. The connected machine was running Linux, but I see the same problem under Windows 7's (to use Media Center) over HDMI:

Here's an image I scaled down by 5%, then back up to original size, which demonstrates a similar effect:

Imagine what this does to text, typefaces, and most kinds of pattern. Argh!

It's just too crap

Before you say "It's a TV, not a computer monitor", let me quote from Kogan's product page for their new model 32" 1080p HD TV, the 1080P-BD32:

32"/ 81cm Full HD panel (1920x1080)
Full High Definition panel ensuring crystal clear TV with Progressive Scan up to 1080P. Doubles up as an excellent computer monitor.

While that text did not appear on the product page for my TV, it shows that Kogan, like me, think that it's reasonable to use modern HDTVs as monitors. In any case, I'm using mine for a lounge room media PC, not a regular PC, and just want it to work! I don't expect amazing quality, but actually displaying the pixels I send it and doing so with working HDMI audio surely isn't too much to ask?

After hours of work and several conversations with Kogan, I finally have the TV running at its rated 1920x1080 (1080p) resolution without horribly butchering the image with overscan. Of course, to get that far I had to override the EDID and disable HDMI audio. If there's no HDMI audio signal the TV plays the 3.5mm input on all HDMI channels - but that means you have only one input for all your channels, making the HDMI combined audio/video feature spectacularly useless, and rendering the TV effectively capable of only one distinct A+V input among from the 2 HDMI and 1 VGA ports. Needless to say, this sucks.

At this point the TV is annoying me enough that I'm planning on gutting my old laptop and using its 1080p capable LVDS output to drive the TV's panel directly, bypassing the TV mainboard. That way I'll be able to see how much of this is the panel and how much is that CultraView mainboard. If the panel is OK, I'll grab a little ARM or Via Nano board with a couple of HDMI inputs and a high-res LVDS output and build that into the case as an embedded media PC, getting rid of the CultraView part entirely.

Well, I've certainly learned one thing. I won't be buying from Kogan again unless I can test the product in person with my equipment before buying. They assemble products from 3rd party parts - which I knew. They don't push issues back to those suppliers and get fixes for them, which I didn't know and didn't expect. Poor show, Kogan.

Are you a programmer+sysadmin+support person at a small company? You need to read this.

2010-10-16T13:41:00.002+08:00

I'm a sysadmin, developer of in-house software, and end-user tech support person at a small company. I wear a lot of hats, and do everything from Java EE applications to data recovery from crashed user laptops. If you can't open that email attachment or get to that website, I'm your man.

Needless to say, this makes the programming side rather challenging. I've always thought of interruption as the enemy of effective programming, and I've observed my effectiveness falling as the rate of interruption increases.

This article gives me reason to reconsider that idea, and consider trying to plan for and work with interruption instead. I cannot recommend it enough.

http://www.stevestreeting.com/2010/09/04/work-2-0/

On SSDs

2010-10-15T11:33:00.004+08:00

I suspect that some of the ideas behind the design of SSD drives as currently sold are rather flawed. Update: See the end of the article for an alternative viewpoint, though.

These drives are embedded computers, with their own CPUs, RAM, firmware, etc. They are much more complicated than they should be for a simple storage device, and are more like mini RAID controllers than they are like hard disks.

The reason things are done this way is because of the dire state of direct flash memory support in most OSes. NTFS, HFS+, Ext3/4, etc are all focused on rotating media. They try to minimize seeks and fragmentation, don't care about grouping writes/overwrites together, etc. For solid-state flash-based storage to work well with these file systems and the OSes that use them, the drives need to do a lot of work behind the scenes to cope with the file system's access patterns.

In an attempt to work around the mismatch between flash storage and rotating-media-file systems, current SSDs also sometimes cheat and do things like claim to have flushed their buffers to persistent storage when they really haven't, or ignore write-ordering barriers, both of which can cause horrifying data corruption for database workloads if power is lost suddenly. (Most SSDs don't suffer from this problem - high-end enterprise drives have big capacitors that give them time to write out their caches before losing power, and most consumer drives respect the OS's request to flush their write cache. Test carefully before trusting one, though.)

Existing flash-oriented file systems like JFFS2 are (a) not widely availible on major platforms, and (b) severely limited in the maximum size of media they can handle, because of the requirement to scan the media to build in-memory metadata before becoming usable. This has led to the release of SSDs oriented toward use with existing file systems - which *can't* be accessed directly for use with intelligent, flash-aware file systems.

That's the bit that drives me nuts. None of the consumer and few of the enterprise SSDs can be lobotomized to present a dumb pass-through direct-access interface for OSes, HW raid controllers, etc that know how to use the flash directly. Because of this, we're stuck with the frustrating situation where file systems focused on rotating media are being optimized to work better on flash storage that's trying to pretend to be rotating media. This makes it hard to build and test, let alone adopt, file systems that're actually designed for flash!

With 64-bit (OK, physically 48-bit) machines, we should really just be able to map the flash storage directly into host address space and let the OS play with it that way. With an IOMMU (universal these days) it can be done safely and securely, and you'd benefit from the host's memory barrier support etc. Alternately, a simple direct PCI-E interface to access the flash wouldn't be too much to ask. This strange way of doing it by pretending to be a SATA hard disk appears to be wasteful, expensive, and SLOW. It also makes it nigh-impossible to do interesting things with flash SSDs, like use them as write cache for software RAID.

It's not showing any signs of getting better, either. You'd think that one or more SSD manufacturers would be working with Microsoft, at least, to provide an NTFS port optimized for direct use on flash, so they could avoid including all that *expensive* non-flash-chip hardware on their SSDs.

Update: Arguably the complexity and abstraction of SSDs is an extension of the existing trend in hard drives. First, HDDs went from electronics-offboard MFM and RLL drives to electronics-onboard IDE drives. Drives later gained logical block addressing (LBA) to hide the drive's internal layout from the OS, so the old c/h/s mappings became meaningless and the OS lost all visibility into the drive's internal layout, having to trust the drive to use a sensible block mapping. Shortly thereafter drives began to transparently remap bad sectors in firmware, so the OS didn't have to be aware of failed sectors or map them at the file system level. Command queuing and reordering arrived a while thereafter, letting the drive make intelligent decisions about the order in which it executes requests. Now modern HDDs even have extensive self reporting and analysis tools like S.M.A.R.T . A spinning-disk HDD is already a small computer, and provides a useful abstraction of storage away from the OS so the OS doesn't have to understand the fine details of every drive.

Viewing things this way, perhaps the complexity of SSDs is a natural evolution. Perhaps the SSD firmware and PLC will simply do a better job of wear-leveling and write-ordering than the host file system ever can. Maybe on-board supercapacitor-protected write-back cache offers features that can't be matched at the host level. Perhaps it's a case of "right tool for the job" and sensible levels of abstraction. Keeping write-back cache close to the storage it is cache for is a particularly compelling advantage, and one that'd be particularly hard to match with host-controlled flash.

Personally, I think it's probably partway between the two. SSDs will have to expose more and more SATA/SAS extensions to allow host OSes and file systems better visibility into their innards and to gain better control. Host file systems will have to become more aware of SSDs and more able to adapt their usage patterns. Perhaps SSDs that plug straight into PCI-E and present an ACHI to the host without any real SATA bus in the middle will become common. Much like RAID controllers, modems, and many other things have slowly moved from hardware to running on the host CPU, so much of the SSD workload is likely to move to the OS driver level over time. On the other hand, a few things like safe write-back caching are probably always going to be better done in the individual SSD rather than the host, much as current ATA drives map logical addresses to C/H/S internally. It's just going to take a while for the division of responsibilities to settle out. The difficulty of plugging new file systems into existing OSes makes it initially more appealing to do everything in hardware, but over time that'll shift as flash gets cheaper and the control hardware becomes a bigger proportion of overall cost, pushing SSD manufacturers to move more and more control logic from expensive hardware into cheap drivers on the host OS.

Notes on gdb use

2010-10-04T11:42:00.004+08:00

I find myself looking up how to do certain things with gdb, because I use it rarely enough not to remember, but frequently enough for these to be somewhat annoying. This post notes how to handle things like signals and paging in gdb.

Disable paging in gdb:

set height 0

Tell gdb not to stop execution on sigusr1 or sigusr2 - good for debugging postgresql:

handle SIGUSR1 nostop
handle SIGUSR2 nostop

BoB (Belkin F1PI243EGau) DNS is still broken

2010-10-04T09:16:00.013+08:00

Update Oct 19, 2010:Contacting Belkin sales and customer feedback and pointing out my increasing efforts to show how poorly they've performed in public finally got a response from Belkin. Admittedly the response is to say they've put the issue though to an "overseas engineer" ... but maybe something will happen. More likely, it'll stay trapped in another layer of disinterest and poor management, this time one I can't apply direct pressure to.

UPDATE Oct 6, 2010:An indirect approach, by bothering a friend who works at iiNet, got this issue through the support wall and to people who can deal with it. Belkin has been notified at a higher level too. It's a real shame that a clearly demonstrable issue like this got stuck behind support people at both companies, to the point where I had to bother a friend who shouldn't have to deal with this stuff just to get the issue looked into. Sometimes tech support acts as a barrier that prevents a company from finding out about real problems, an issue I've seen not only with iiNet and Belkin but with endless other companies. Anyway, hopefully Belkin will be getting onto this now.

A year ago, I reported to iiNet that AAAA lookups in the BoB (F1PI243EGau) DNS forwarder were always timing out, rather than returning SERVFAIL or correctly forwarding the query to the upstream server. This is the cause of the slow browsing issues reported for the BoB.

A couple of months ago I got hold of a pre-release firmware that fixed this, and about a month ago the firmware was finally put up for public use. This fixes the AAAA issues, so browsers like Firefox and Safari on IPv6-capable operating systems like Mac OS X and Windows 7 don't take an eternity to resolve every DNS query.

Unfortunately, Belkin didn't take this as a hint to properly test their DNS forwarder. They fixed AAAA lookup, but didn't fix it to return SERVFAIL when it encountered something it didn't understand, and failed to test other record types like TXT and SRV.

Sure enough, TXT and SRV lookup have the same problem. This is currently causing problems with Google Talk (using Pidgin) that require the configuration of a fallback connect server to bypass TXT record lookup.

Belkin support do not understand the problem. The low-level support folks at iiNet don't seem to get it either. Neither are passing the problem on to somebody with the experience and knowledge to understand the problem, and neither seem to have access to suitable hardware - or the inclination to use it if they do - to verify the issue.

Here's the explanation I sent to them.

The BoB (Belkin F1PI243EGau) doesn't handle DNS queries for TXT and SRV records correctly. Instead of forwarding these queries correctly or even returning SERVFAIL, it fails to respond, causing a client-side timeout.

Before the latest firmware update it had the same problem with AAAA (IPv6 DNS) lookups.

This issue causes problems with Google Talk (why I noticed the problem), Active Directory, SPF mail lookups, etc. It's particularly obvious when using Pidgin for Google Talk, as it relies on correct SRV record handling, where the official Google Talk client falls back to "brain dead router" mode hard-coded defaults if SRV lookups fail.

I won't write a whole document on what TXT and SRV records are and why they're important here; that's what Google is for. What I will do is provide you with instructions on how to demonstrate that the BoB's dns resolver doesn't handle them correctly while other DNS resolvers do.

I'm going to assume you're on Windows. If you're on Mac or Linux, replace "nslookup -a=txt" with "dig +short -t txt" ; otherwise the commands are the same.

In a command prompt window (start>run->cmd.exe) on a machine that has access to a BoB (F1PI243EGau) with IP 10.1.1.1, run:

nslookup -q=srv talk.google.com 10.1.1.1
nslookup -q=txt gmail.com 10.1.1.1

Both are queries for valid, existing records. You will find that those requests both time out:

C:\Users\Craig>nslookup -q=txt gmail.com
Server: bob.iad
Address: 10.1.1.1
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to bob.iad timed-out

C:\Users\Craig>nslookup -q=srv talk.google.com
Server: bob.iad
Address: 10.1.1.1
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to bob.iad timed-out

Now run the same commands again, but this time use a real DNS server directly, bypassing the BoB. Because I have no way of determining what your local DNS server IPs are, I'll give you an example that uses Google's public DNS directly, but you'll find it works with any DNS server other than the BoB:

nslookup -q=srv talk.google.com 8.8.8.8
nslookup -q=txt gmail.com 8.8.8.8

( 8.8.8.8 and 4.4.4.4 are Google's public DNS servers ).

Note that they succeed, because we're bypassing the BoB's broken forwarding DNS resolver?

C:\Users\Craig>nslookup -q=txt gmail.com 8.8.8.8
Server: google-public-dns-a.google.com
Address: 8.8.8.8
Non-authoritative answer:
gmail.com text =
"v=spf1 redirect=_spf.google.com"

C:\Users\Craig>nslookup -q=srv talk.google.com 8.8.8.8
Server:  google-public-dns-a.google.com
Address:  8.8.8.8
Non-authoritative answer:
talk.google.com canonical name = talk.l.google.com
l.google.com
        primary name server = ns3.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 1429061
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)
        default TTL = 60 (1 min)

As you can see, the BoB's forwarding DNS resolver isn't correctly handling TXT and SRV record lookups. Note that this persists even if I override the ISP-configured DNS servers and use Google's instead, so it's not a problem with my ISP DNS. In any case, queries directly to my ISP DNS work fine. TXT and SRV queries fail from my Windows 7 desktop, my Linux laptop, and spare Mac OS X machine I borrowed; this is not a problem specific to one computer or operating system, it's a firmware issue.

For bonus points, here are the brillant replies from the Belkin support staff:

Dear Belkin User,

Thank you for contacting Belkin Technical Support.

We understand your concern and We will be happy to assist you with your queries.

In order to isolate the issue we suggest you to update the firmware on the router.

Please find the link below where you can update the firmware for the router F1PI243EGau. http://en-au-support.belkin.com/app/answers/detail/a_id/2703/session/L3NpZC8xUHpIcm9iaw%3D%3D/kw/f1p1241egau/r_id/166/sno/0

(Editorial note: this link refers to the firmware for the wrong router, the F1PI241EGau, so clearly even Belkin can't keep track of their model number scheme)

Please do feel free to write back if you have any further queries,we will be more than happy to assist you.

When I replied to ask if they really meant that firmware, and pointed out that my original report showed that I was already running the latest firmware for my router, the F1PI243EGau, they replied with:

Dear Belkin User,

Thank you for contacting Belkin Technical Support.

We understand your concern and We will be happy to assist you with your queries.

(Look like another template answer to you? Me too.)

We would appreciate if you please get back to us with the following information to isolate the issue:

Are you trying to do a NAT loop back?

Are you trying to use DDNS?

Please do feel free to write back, and we will be more than happy to assist you. Please take a moment to review our Knowledge Base at http://belkin.com/support or let us know.

After this brillant uttering, they went silent and stopped responding to further requests. Belkin's support for their product as demonstrated here is an abject, total failure.

Exception breakpoints

2010-09-08T01:38:00.002+08:00

Do you have a StackOverflowError thrown deep inside some (probably 3rd party) code like EclipseLink's Criteria API? Are you trying to track down exactly where it's being thrown and under what circumstances? Is it driving you insane?

Use an exception breakpoint. In Netbeans: Debug -> New Breakpoint. Breakpoint type "Exception". Specify the exception class being thrown, constrain the throwing class if desired, and you're set.

This made my life a lot, lot, LOT easier recently. Thanks again NetBeans.

Java EE apps as first-class citizens

2010-08-29T19:50:00.003+08:00

Juliano Viana writes that he hopes to see Java EE applications promoted to first-class citizens in Java EE 8.

His comments are focused on protecting apps running in a container from each other, and the container from the apps. The primary focus of the post is on resource leaks. He's interested in making it possible to cleanly and totally unload an app, much as you can kill a process on a regular OS to completely clean it up.

This lead me to an interesting realization, though one I'm sure others have had before:

Java EE containers are in some ways a lot like MS-DOS, Mac OS 6/7/8/9, Symbian, etc ... applications share the same memory space and have access to each other's innards. There's polite agreement not to poke where they don't belong, but no strong enforcement of that. Without pointers and the resulting fun with corrupted stacks and dangling pointers leading apps to trample all over each other's memory the JVM and Java EE don't have it quite so bad - but there are still real issues with access control, resource ownership, etc.

Currently a Java EE app can delve into the innards of the container if it wants. It's not *meant* to, but it's not easily prevented and it makes the container's integrity less trustworthy. An app can break the container or cause it to misbehave. Expecting apps to be well-written is, as demonstrated by the OSes listed above, a great way to get a reputation for being unstable and generally crap. More importantly, as the referenced article notes, Java EE apps can cause the container to leak resources, or can disrupt resources the apps shares with other apps via the container, like pooled connections. That makes it hard to share one container among many different apps, let alone different users/customers.

This (and garbage collection + resource use issues with big JVMs) makes admins reluctant to have many apps hosted in a single container, so you land up doing all sorts of icky inter-JVM work to make your environment managable. Shared-hosting services with Java are expensive and uncommon due to related issues.

I certainly think that much stronger protection is needed to isolate the container's private core from apps, and apps from each other. I'm not at all convinced that trying to build that isolation entirely in software in the JVM is best, though. Modern successful OSes isolate apps using *hardware* features to do most of the grunt-work and protect them from implementation bugs; apps and the kernel are isolated from each other by default and can only interact by configured exception. Protected mode, separate logical address spaces, and the security ring system in x86 provide rather strong OS/app isolation. Trying to implement similarly capable isolation purely in software in the JVM would be bug-prone and hard to get right, kind of like the half-baked and buggy app isolation in Windows 3.x or even 9x.

Perhaps a more viable future approach would be to let multiple JVMs running on a host integrate more tightly via shared memory, high performance IPC, or whatever, so the container could run in its own JVM and each app could have a private JVM, with only those resources *explicitly* shared between them accessible across JVMs. That way, apps cleanup would be as simple as killing the app's jvm and letting the OS+hardware clean up the mess. The exposed surface for problems would be restricted to that part of the container that's explicitly accessible via the shared resources.

Or ... maybe the JVM will need more OS co-operation than that, so the JVM its self can use hardware features to isolate apps running within the JVM. I can't imagine OS designers wanting to let the JVM get its hooks into the kernel's memory mapping and process management, but with the advent of VMs and VM-friendly CPU extensions like VT-X and VT-IO I wonder if the JVM could use those instructions, like the kernel of a guest OS does in a VM, to isolate apps running within the JVM.

Much as I'd love the JVM to be able to isolate apps properly, in my admittedly pretty ignorant opinion I don't see it happening without some kind of major re-design. I imagine a half-assed solution that works about as well as Windows 9x did is possible with the current JVM, but giving the impression apps can be isolated and cleaned up without being able to really do a comprehensive job of it is IMO in many ways worse than the current situation. Without some really powerful, low level isolation features I don't see that happening. Look at how well "shared" OSes have typically achieved app isolation for examples of just how hard the problem is.

I find the idea of using hardware VT-X support to help the JVM isolate apps in an EE container quite intriguing. I wonder if it's something anyone's done any investigation of.

Three hours in

2010-08-26T16:25:00.005+08:00

I'm three hours or so into my real work, after being side-tracked isolating and reporting Glassfish and Mojarra bugs. I just found another one in Mojarra, albeit a trivial one.

(Edit) I'd written a little rant about the problem I was trying to solve, in the assumption that I'd hit another limitation/bug in JSF2. Well, this time I was just plain wrong; I'd used an incorrect test to draw an invalid conclusion and from there ran down the path - from my previous experience, admittedly - of assuming that this, too, was a tools issue.

The correct answer is here: To permit null/empty/none selection in a h:selectOneMenu driven by a f:selectItems , just add a f:selectItem with no value before the f:selectItems.

Converting a NetBeans-generated JSF2 project to use CDI

2010-08-25T17:29:00.010+08:00

I started playing with Netbeans 6.9's JSF2 interface generation for database apps, and quickly noticed that (a) it uses JSF2 injection, not CDI and (b) it generates some verbose, ugly converters.

I decided to switch it over, as a bit of a learning experience. Here's what landed up being necessary.

Switching the controller beans to CDI injection is as simple as changing:

import javax.ejb.EJB;
import javax.faces.bean.ManagedBean;
import javax.faces.bean.SessionScoped;

@ManagedBean
@SessionScoped
class AdController {
   @EJB SomeFacade ejbFacade;
}

to:

import javax.inject.Named;
import javax.inject.Inject;
import javax.enterprise.context.SessionScoped;

class AdController implements Serializable {
   // Use a uid generator to provide a suitable id here:
   private static final long serialVersionUID = 1L;

   @Inject AdFacade ejbFacade;

}

In other words, we've switched from JSF2 @EJB injection to CDI @Inject, from JSF2 @ManagedBean to CDI EL name annotation @Named, and from the JSF to the CDI scope annotation for @SessionScoped (they're in different packages but have the same name). Because CDI enforces the requirement that session beans be serializable, we implement serializable - though in practice this was necessary for correct session bean passivation in JSF2 even without CDI.

There's a catch in this conversion - the static inner converter classes in each of the controller classes.

Injection doesn't work in a @FacesConverter* so you can't @Inject the EJB facade you want directly into your converter. That's why the converter code generated by NetBeans uses such a roundabout method to get the ejbFacade, by looking up the controller bean in EL and getting it from there. However, CDI clears injection sites between phases. Because CDI has cleared the ejbFacade member by the time the converter runs, it'll be null by the time we need it in the converter. You're likely to land up with null pointer exceptions due to null ejbFacade members in the controller beans the converters look up up via FacesContext. (See this Mojarra bug)

To work around that, we'll use the fact that the Netbeans-generated facades share a common abstract base to write an abstract base for the converters. This base will know how to look up an EJB from CDI directly, without relying on injection. Here's the code, which is derived from Dominik Dorn's example:

package com.mycompany;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;
import javax.faces.convert.Converter;

public abstract class AbstractFacesConverter<EntityType, FacadeType extends AbstractFacade<EntityType>> implements Converter {

    private final Class<FacadeType> facadeClass;
    private final Class<EntityType> entityClass;

    protected AbstractFacesConverter(Class<EntityType> entityClass, Class<FacadeType> facadeClass) {
        this.facadeClass = facadeClass;
        this.entityClass = entityClass;
    }

    @Override
    public Object getAsObject(FacesContext facesContext, UIComponent component, String value) {
        if (value == null || value.length() == 0) {
            return null;
        }
        FacadeType facade = (FacadeType) CDIHelper.getManagedBean(facesContext, facadeClass);
        return facade.find(getKeyFromString(value));
    }

    @Override
    public String getAsString(FacesContext facesContext, UIComponent component, Object object) {
        if (object == null) {
            return null;
        }
        if (entityClass.isInstance(object)) {
            return getKeyAsString( (EntityType)object);
        } else {
            throw new IllegalArgumentException("object " + object + " is of type " + object.getClass().getName() + "; expected type: "+AdController.class.getName());
        }
    }

    /**
     * Given the string-form of the primary key for the target object,
     * return a representation of the primary key in the proper type.
     *
     * @param value String representation of primary key
     * @return Primary key
     */
    protected abstract Object getKeyFromString(String value);

    /**
     * Given the an entity, return a string representation of the
     * primary key of that entity. The string must round-trip back
     * to the PK of the entity when passed to getKeyFromString().
     *
     * @param entity Entity to get the PK of
     * @return A string representation of the PK
     */
    protected abstract String getKeyAsString(EntityType entity);
    
}

... and here's a converter for an entity with an Integer ID that's been implemented using the abstract base converter:

    @FacesConverter(forClass=Ad.class)
    public static class AdControllerConverter extends AbstractFacesConverter<Ad,AdFacade> {

        public AdControllerConverter() {
            super(Ad.class, AdFacade.class);
        }

        @Override
        protected Object getKeyFromString(String value) {
            return Integer.valueOf(value);
        }

        @Override
        protected String getKeyAsString(Ad entity) {
            return entity.getId().toString();
        }

    }

Much nicer, eh?

Let's try running the code. It should work initially, but if you're on Glassfish 3.0.1 it'll fail with an exception like:

java.lang.IllegalStateException: Unable to convert ejbRef for ejb AdFacade to a business object of type class com.mycompany.AbstractFacade

... as soon as you try to do anything much. The cause is this Glassfish/Weld integration bug. As a workaround for the bug, you can override all the methods implemented in the AbstractFacade in each concrete facade EJB with trivial wrapper methods. NetBeans' "Insert Code" -> "Override Method" context menu option is very useful for this.

Once you've wrapped your inherited EJB methods, you should finally be back where you started, with an app that's now CDI-enabled and does exactly what the JSF2-only one did. Yay?

* The SeamFaces project - a part of Seam 3 - is intended to fix that, but isn't ready yet. There are also apparently plans to fix this limitation in JSF 2.1. Seam Solder offers tools for BeanManager lookup, etc ... but doesn't work on Glassfish 3.0.

Untangling Java EE 6 - a broad conceptual overview

2010-08-24T15:17:00.016+08:00

WARNING

This document was written by someone who's only learning Java EE 6. On one hand, I remember the frustrations clearly and still encounter issues of understanding (and bugs) regularly, so I'm in a good position to think about how a newbie needs to have things explained. On the other hand, a bunch of this is probably wrong because my own understanding is still developing. Beware.

This is also a work in progress. It's been published for comment and review, and isn't a fully edited and checked final work.

If you're coming from Spring, rather than starting with Java EE cold, you should start here.

Java EE 6

Java EE 6 is not a product you can download and install. It is a specification that describes how implementing software should behave. It refers to many sub-specifications for particular features, each of which has one or more implementations in software. Some of these implementations can be used outside a full Java EE 6 environment as standalone products.

There are a few different application servers availible that provide full Java EE 6 environments by collecting and integrating the implementations of the various Java EE 6 components into a product. These products have their own features, administration tools, etc as well as support for the Java EE 6 spec.

Unsurprisingy, when you are trying to get a handle on how it all works, it's very confusing. The fact that most Java EE 6 documentation and tutorials refer to Java EE 5 with the assumption that you're an existing Java EE 5 user upgrading to Java EE 6 doesn't help.

Here I'll try to provide an overview of how it all fits together, helping you decode the names and figure out what's what.

Java EE 6 isn't a downloadable, installable product

The first thing to understand is that Java EE 6 isn't a piece of software. It's a client interface specification (API) describing what services an environment (the "container" or application server) provides to software running within that environment, and how those services should behave. It's also a server specification that partly specifies now those services should work and what they should do.

Consequently, you can't "install" Java EE 6, you can only install an implementation of it like an application server.

There is one piece of concrete code for Java EE 6: the API JARs. The Java EE 6 api and Java EE 6 Web Profile API jars contain Java interfaces that let you compile code to use Java EE 6 services and features. They do not contain implementations of those features, and do not require a particular implementation; that's the job of the particular Java EE 6 container/app server you use.

The parts of the Java EE 6 spec

As already noted, the Java EE 6 spec is composed of many parts.Some of them will be familiar, like JavaMail and JDBC, while others like EJB 3.1, CDI, JNDI and JAX-RS will be unfamiliar to you. Java SE is availible in the Java EE environment, so most of what you are used to in Java SE should remain unchanged, but the way you use it is completely different.

There is a list of the technologies used in Java EE 6 here:
http://www.oracle.com/technetwork/java/javaee/tech/index.html
Most of those technologies can be used outside a full Java EE 6 environment, and were developed as individual specifications under the Java Community Process (JCP), usually alongside a reference implementation of the specification being developed.

Application servers (containers) that support the Java EE 6 spec promise to provide implementations of all the various sub-specs in Java EE 6 for an application's use.

Application servers and containers

Application servers like Glassfish and JBoss AS implement the Java EE 6 spec, providing applications with a managed container they can run in to use all the Java EE 6 services. You can think of the container as a lot like the operating system and JVM a Java SE applicatoin runs within - it's outside the application, but provides services and facilities that enable the application to work.

The app server takes responsibility for making the spec's requirements happen, so user code just has to use the Java EE 6 interfaces.

There are also containers like Tomcat and Jetty that do not implement the full Java EE 6 spec. It is possible to use most of the technologies used by Java EE 6 within such containers by installing implementations of them in the container or bundling them with your application. This is why you will see references to using Tomcat or Jetty with technologies like JSF2 that are also part of Java EE 6.

The vendor of a full Java EE 6 applicaton server like Glassfish or JBoss AS application server vendor doesn't implement all of Java EE 6 directly either. Most of the Java EE 6 implementation provided by an app server is in separate products bundled with the server to provide an implementation of each individual Java EE 6 sub-spec. The server vendor chooses the implementation to use for each sub-spec, and provides the code to load them and integrate them into the full environment. Some specs (like EJB 3.1) will be implemented directly by the server vendor and won't be a separate product or package, though they'll generally still be a module within the application server.

For example, Glassfish 3 (an application server) supports java EE 6 (the spec), including support for CDI (a sub-spec) using JBoss Weld (an implementation of the CDI spec). Glassfish also supports EJB 3.1 (another sub-spec) as a core part of the Glassfish server, rather than via 3rd party code.

You will have noticed that Glassfish uses a component, JBoss Weld, from a competing application server vendor. That's commonplace, so you shouldn't assume that because a component was developed by Oracle, Apache, JBoss, etc it's only used in their application server.

A list of servers, specs, and implementations of those specs can be found at the end of this article.

Using Java EE 6 in applications

Applications written for Java EE 6 are compiled and bundled into a package (an ejb jar, a war, or an ear file: see later) that is deployed to the application server. Deployment involves loading the application archive after copying it to the server if required, scanning it for deployment descriptors like web.xml, and setting it up in the container's environment so it can be accessed. Depending on the application, that might involve enabling web service listeners, adding servlets to the container's HTTP namespace, etc.

Applications do not have to use all the features of Java EE 6, and can choose to only "see" features they use. Instead of having the full Java EE 6 API JARs on the compile-time classpath, they can include only the API JARs for the sub-specs they want to use. For example, an application might choose to use the Servlet API and the JPA 2.0 API, but not bother with the EJB 3.1 API or JavaServer Faces. To do that, it omits the full Java EE 6 API JARs from the classpath, and instead includes the API JARs for the sub-specs of interest. You will see this a lot in example code - for example, JavaServer Faces (JSF) projects often use the JSF API and Servlet API jars rather than the Java EE 6 API jars.

Even if you include the full JAva EE 6 API JARs, many features won't affect you (or activate at all) unless you use them by putting their annotations in your code, configuring them in web.xml, etc.

Because you don't have to use all of Java EE 6, it's quite possible to build an application that'll deploy and run in regular containers as well as full Java EE 6 application servers. Such an application generally requires additional software to be installed in the non-EE container to support the features it needs, or has to bundle the implementations with the application.

Java EE 6 can be extended

Just as you don't have to use all of Java EE 6 in applications, it's common to add additional components to extend the capabilities of the application server. Some of these components are regular class libraries, but some of them are container components that use servlet filters or other EE features to extend the capabilities of the container its self.

For example, instead of using the Java EE 6 standard web interface toolkit, it's fairly common to use one of the innumerable other toolkits like Wicket, Tapestry, Struts, etc instead, while still running within a Java EE 6 application server container.

It's also common to include extensions to Java EE 6 functionality. For example, there are JSF 2.0 extensions like RichFaces and PrimeFaces that add new features and new components to the existing JSF 2.0 implementation.

One key extension is SeamFaces, which seeks to address some of the more painful limitations of JSF2, like the inability to inject resources into a @FacesConverter. SeamFaces is a work in progress and wasn't at release quality at time of writing; see the project page for details on its status.

Pitfalls

There are some holes in the current Java EE 6 spec, and some areas of confusion where the required behaviour isn't particularly clear. These will be discussed in detail in separate documents.

@FacesConverter can't be a managed bean and isn't subject to injection. See SeamFaces (above).
Multiple dependency injection implementations exist in the core spec.
- JSF2 provides basic dependency injection support, liftime and scoping, EL naming, etc. These live in the javax.faces.bean package.
- The CDI spec provides a more complete dependency injection (@Inject), lifetime and scoping, and EL naming system using different facilities that *mostly* interoperate with JSF2's. The annotations live in the javax.enterprise.context and javax.enterprise.inject packages.
- EJB 3 has its own injection via @javax.ejb.EJB
The JSF2 features are not clearly deprecated and it's not always clear how they will interact with CDI or old-style EJB interaction. A separate document will discuss CDI and JSF2 injection and scoping.

There are also a few things that, while not JSF issues as such, are likely to bite you if you don't know about them, and aren't particularly well documented:

@PersistenceContext(type=PersistenceContextType.EXTENDED), ie an extended persistence context, can only be used within a @Stateful session bean. In other contexts, like a POJO, you'll get odd errors from within the runtime, like (in Glassfish 3) a NullPointerException from com.sun.enterprise.container.common.impl.EntityManagerWrapper.

Thinking Java EE 6

There are some really big differences between how a Java EE 6 application is constructed and how a Java SE application is constructed. Many of these stem from the container-managed nature of Java EE, where you provide a set of interacting components that run in the container, rather than a whole "program" with dedicated flow control from start to finish. Adapting to Java EE is a lot like adapting to event-driven programming for someone used to the imperative programming model - it's brain-bending, but worth it. You have to relax the idea that you dedicate the flow of execution through your code, and instead get used to providing a set of services that interact via events, messages, and the actions of client code.

Even object lifetimes and object instantiation are largely out of your hands. Most of your objects will be container-managed, with lifetime and scope controlled by the container and linked to the user request or session. The objects they interact with will be injected, rather than obtained from a factory or directly created. It's the container that's responsible for selecting or creating an instance of an object to inject, not your code.

You will hear terms like "dependency injection" and "inversion of control" thrown about, but they won't mean much to you until you start using Java EE in practice. Once you wrap your brain around it then, just like event-driven programming, it becomes an obvious and sensible way to do things and lets you just forget about problems you used to obsess about. Mostly, anyway.

Where to go from here

Start with the excellent Java EE 6 tutorial. Don't forget about the Java EE 6 API documentation and the other resources availible on Oracle's Java EE 6 page either.

It can be helpful to grab the spec for a given Java EE 6 technology from the JSRs linked from here.

There's a good tutorial on JavaServer Faces here. Avoid the JSF specification, as unlike most of the surprisingly well written JSR specs it's an unreadable monstrosity that will only confuse you.

Names and acronyms galore

One of the big challenges of understanding Java EE 6 is figuring out what all the names, acronyms and code annotations mean and how the inter-relate.

This short guide will attempt to untangle things for you. (It's currently very incomplete, though, and should probably move to a separate post.)

Containers
- full Java EE 6 app servers
-- Glassfish 3
--- Uses Mojarra (JSF2 and JSP), Weld (CDI)
-- JBoss AS 6 (prerelease)
--- Uses Mojarra (JSF2 and JSP), Weld (CDI)
- Java EE 5 containers
-- Glassfish 2
-- JBoss AS 5
-- Geronimo (built on Tomcat / Jetty)
-- WebSphere
-- WebLogic
-- Etc etc. See wikipedia "comparison of application servers"
- non-EE applicatoin containers / servlet engines
-- Tomcat
-- Jetty

Technologies used in Java EE 6 with widespread competing implementations
- CDI
-- JBoss Weld (RI)
-- Apache OpenWebBeans
- JSF2
-- Mojarra (RI)
-- Apache MyFaces
- JPA2
-- Apache OpenJPA
-- Oracle TopLink Essentials
-- JBoss Hibernate 
-- EclipseLink

Commonly used add-ons
- JSF2 extensions and component libraries
-- RichFaces
-- IceFaces
-- PrimeFaces
-- PrettyFaces
- Non-JPA persistence
-- Direct use of JDBC
-- iBatis / MyBatis

Other names to place:
- Apache Felix (OSGi)
- JBoss Seam 2 and 3
- Spring

Other references
- Wikipedia "Java EE version history"

A separate post will be dedicated to associating commonly used EE annotations with the specifying JSRs and implementations of those JSRs. Things like @ManagedBean, @Named, @SessionScoped, @RequestScoped, @ApplicationScoped (both CDI and JSF versions), @Local, @Remote, @Stateless, @Stateful, @Singleton, @Inject, @Resource, @EJB, @PersistenceContext, etc. Particular attention needs to be paid to areas of overlapping functionality and the differences between the options: JSF scoped beans vs CDI scoped beans vs EJBs in lifetime management, for example.

Java EE 6 doesn't "just work" in even trivial real-world uses

2010-08-22T14:14:00.007+08:00

Lincoln Baxter III writes that with Java EE 6, It all “Just works”.

Personally, I have not found that to be the case in even simple, toy applications.

I suspect it'll be true in a few years, probably after a much needed "Java EE 6.2" and some major bug-fixing. Right now it all "just works" ... except when it doesn't, which is frequently, due to spec oversights, bugs, and a general lack of real-world use and testing.

Using CDI "just works" until you realize that @Inject isn't supported in obvious places like a @FacesConverter due to limitations in JSF2/CDI integration. OK, that's easily worked around by looking up a managed bean from the FacesContext and getting the desired EJB from the @Inject site on that managed bean. Except that the ejb injection site is null during that phase of processing, having been helpfully cleared by CDI, so you get a null pointer exception that looks downright inexplicable until you figure out that CDI is clearing the injection site behind your back. (See: this report and this Mojarra bug). In that case I eventually found out how to look up the desired EJB via the BeanManager (See: http://dominikdorn.com/2010/04/cdi-weld-manual-bean-lookup/) but it certainly doesn't "just work" even between ostensibly co-operating, integrated technologies.

I hit that roadblock *after* getting stuck on a Glassfish/CDI integration bug (https://glassfish.dev.java.net/issues/show_bug.cgi?id=13040) that means that those nice new no-interface views of local EJBs don't actually work if you inherit methods from a superclass and are using Glassfish (even 3.1 as of build 15).

To add to the confusion, often using the JSF2 injection support to do something works, but CDI doesn't, or vice versa. And where the JSF2 or CDI injection support works on Glassfish it'll often fail on the (admittedly alpha) JBoss AS 6 or vice versa.

This is in my first JSF2 project on a Glassfish 3.0.1 container, using all the new "just works" stuff, and I must say that it sure as hell doesn't "just work" some of the time.

Another factor that doesn't help is all the legacy surrounding Java EE / J2EE. It's hard for someone trying to learn Java EE 6 to separate it from all the JE22 4 and Java EE 5 stuff. Even within Java EE 6 there's "new legacy", things that are effectively deprecated even though they've barely been released. Consider JSF2 scope annotations (replaced by CDI), JSF2 @ManagedBean (replaced by the Managed Beans specification, then promptly replaced again by CDI, all within the span of the Java EE 6 spec) and JSF2 injection support (replaced by CDI). It takes a great deal of sorting out before you can have much idea what the real, practical difference is between:

@javax.faces.bean.ManagedBean
@javax.faces.bean.SessionScoped

and:

@javax.inject.Named
@javax.enterprise.context.SessionScoped

let alone @javax.annotation.ManagedBean (hooray pointless duplication).

Then there's un-muddling the chaos of @Inject, @EJB and @Resource (which I'm still struggling with, since @EJB seems to be used by CDI in a *completely* *different* context), the difference between injecting an EntityManager via @PersistenceContext vs @Inject, etc.

No, it really, really doesn't "just work". It's a nightmare of overlapping, muddled conflicting specs and half-integrated half-co-operating implementations with weird quirks and holes in their support. It really needs an EJB 6.2 release that clearly deprecates all the old stuff and documents how it all interacts.

I've finally got to the point, in my efforts to learn about Java EE 6, where I'm starting to see the bigger picture of how it's *supposed* to work, and supposed to fit together. I can see now how it's meant to be simple to use, and if you don't hit any of the exciting bugs it even can be simple to use and nice to use.

The biggest single improvement that could be made to Java EE 6 is the heavy use of the @Deprecated annotation on 80% of the existing public interfaces, and the option to deploy an app in a CDI-only, no-JSF2-injection, legacy-free environment where attempts to use legacy container services are detected as errors (in development mode) or are ignored/fail (in production mode).

It'd also help to have some centralized documentation that's not focused on one particular implementation, framework, etc, and clearly takes account of spec versions so as to disambiguate discussion about J2EE4, Java EE 5, Java EE 6, and Java EE 6.1.

Simple Java EE 6 project fails in different ways on Glassfish and JBoss AS 6

2010-08-18T17:05:00.004+08:00

(Update: this issue was caused by a Glassfish bug. See Bug 13040 in the Glassfish tracker.)

I'm coming to really like Java EE 6 ... or at least, I would if anything worked properly. There seem to be an awful lot of quirks and bugs in even fairly basic functionality, though.

Take this simple project, where a @Stateless EJB that inherits from an abstract base is injected into a regular JSF2 backing bean. It's done two different ways - once using the new CDI/Weld injection facilities, and once with old-style JSF2 injection and EJB container management.

One works on glassfish, the other on JBoss. Neither works on both.

The glassfish failure is caused by a problem resolving methods inherited from superclasses in the Weld-generated local no-interface EJB proxies. The JBoss issue appears to be caused by issues with its local no-interface EJB support, too, though I'm not sure exactly what yet.

The point: Local no-interface EJBs were supposed to save time and pain for developers, yet all I'm finding so far is bugs, or at least quirks and inconsistent behaviour.

Sources to the sample can be found here and a deployable war is here.

Posted on StackOverflow and The Glassfish forums

Linux (ubuntu 10.04) ppp maxfail 0 and persist not working? (WORKAROUND)

2010-08-16T10:58:00.002+08:00

I've been wondering for a while why, after an upgrade to Ubuntu 10.04, the ppp daemon on my router/server had stopped automatically reconnecting after it lost ADSL line sync or otherwise had its connection dropped.

The culprit turns out to be a helper script (/etc/init/network-interface.conf) in upstart that tries to handle hotplugging of network interfaces, automatically bringing them up when plugged in and down when unplugged.

Unfortunately, it views ppp interfaces as hot-plugged, and helpfully calls ifdown on them when they dissappear after a connection is lost. This kills pppd, preventing it from creating a new ppp interface and trying to connect again.

Unlike most interfaces, ppp interfaces are created and destroyed as a consequence of ifup and ifdown calls. Well, really pppd invocation, but that's usually done via ifup/ifdown these days. Calling ifdown when a ppp interface vanishes isn't always wrong, but it's wrong if the pppd is set to "maxfail 0 persist".

As the script isn't smart enough to know that, and I don't hotplug interfaces on my router anyway, I've opted to simply disable the script in question by renaming it to /etc/init/network-interface.conf.disabled. That'll be broken whenever the upstart package is updated, though, so a better solution is required.

Unconditionally ignoring all ppp interfaces in the script isn't necessarily right, as it would be nice to ifdown them cleanly when the pppd has exited and is thus no long retrying. It's hard to do that reliably as the interface is destroyed before pppd actually terminates, and it's hard to query pppd to find out if it plans to quit or retry. Arguably the retry logic should be moved out of pppd and into upstart or network-manager, but with the state of those tools at this point that's a recipe for pain and suffering.

In the mean time, you can work around it by disabling the problematic init script.

simple script to extract Stanza epub ebooks from an iTunes backup

2010-08-15T21:40:00.002+08:00

iTunes and the iPhone don't like to let go of data they have their claws into. Sometimes, however, you might need that data outside the Apple Garden, in which case you're going to have to get your hands dirty diving in iTunes backups, as they're the easiest way to regain control of your files.

(Tip: if you need to fight your software to access your own files, your platform is hostile. I don't use the iPhone myself and loathe the way Apple does things, but sometimes I have to work with it for other people. Hence this post.)

I needed to recover some Stanza ebooks from an iPhone. It's hard enough to get them *on* to the phone, and getting them off is nigh on impossible, as Apple continuously changes things to make it hard to access the phone via anything but iTunes. In this case, though, the latest change (to the backup format) made it easier to work with, not harder, so the extraction wasn't too hard.

Thankfully, the newer iTunes backup format isn't too hard to work with. It saves two backup files for every file on the phone - one with a .mdinfo extension that's an Apple binary p-list file containing the file's path and metadata, and a second with a .mddata extension that's the actual file data. Other than file extension both have the same name, so they're easy to associate.

The plist format is unpleasant to work with and I haven't made any effort to parse it properly. If anyone has a decent plist parser that doesn't require distribution of Apple shared libraries, please let me know. It's relatively easy to hack together a "dumb" processor that looks for and extracts strings within the plist-format .mdinfo files to obtain path information, as I've done here, but it's extremely fragile and likely to break at minor point revisions of the backup format.

Take the following script, which looks for .mdinfo files containing the string .epub, extracts the filename, and copies the associated .mddata file to the "ebooks" folder in the user's home directory.

Run this script after cd'ing to your iTunes backup folder. On Windows that's in %APPDATA%\Apple\MobiSync (I think, I'm not at the Windows box right now). Within there are folders for each backup. Check the dates to find the most recent, cd into it, and run this script. Your ebooks should appear in a ebooks folder in your home directory.

This is the dirtiest possible hack for the .mdinfo reading and filename extraction. I need to tidy it up into something a bit less gross, but hey, it's enough to play with.

#!/usr/bin/env python
import os
import shutil
import errno

outfolder=os.path.expanduser(r"~/ebooks")

def getfn(f):
    info = open(f,"r").read()
    if "epub" in info:
        return info.split("^P")[2][11:-6]

for f in os.listdir("."):
    try:
        os.mkdir(outfolder)
    except OSError,e:
        if e.errno != errno.EEXIST:
            raise e
    if f.endswith(".mdinfo"):
        fn = getfn(f)
        if fn is not None:
            datafile = os.path.basename(f)
            shutil.copyfile(datafile, os.path.join(outfolder, fn))

I'm not smart enough to use Java EE

2010-08-09T11:27:00.009+08:00

I don't seem to be smart enough to use Java EE. (Update: I seem to be managing anyway, and am trying to help others avoid the same suffering I went through by providing some overview level documentation on how it all fits together and what all the damn acronyms mean.)

You know what helps, though? Reading the manual!, which is actually very good. Before sitting down for some quality reading time, I was totally lost. This is not something you can dive head-first into.

Seam. Struts. Tapestry. Weave. Weld. JSF. JSF2. JPA. JPA2. EJB2. EJB3. CDI. Hibernate, EclipseLink, etc. J2EE. JARs in OSGi bundles in WARs in EARs. Glassfish. JBoss. Tomcat. Jetty. @Inject. @EJB. @Resource. @PersistenceContext. @javax.annotation.ManagedBean. @javax.faces.bean.ManagedBean. @Named. @Stateless. @Stateful. Spring. Servlet filters. @javax.faces.bean.RequestScoped, javax.enterprise.context.RequestScoped, @javax.faces.bean.SessionScoped, javax.enterprise.context.SessionsCoped, @javax.faces.bean.ApplicationScoped, javax.enterprise.context.ApplicationScoped, javax.enterprise.context.ConversationScoped, @javax.faces.bean.ViewScoped. @Singleton. DAOs and data access layers. JTA. JNDI. JMS. RMI. Deployment descriptors (web.xml, etc). Vendor deployment descriptors (sun-web.xml, sun-resources.xml, etc). Various config files, some of which have an effect even when empty (faces-config.xml, beans.xml). Maven plugins, oh god the maven plugins. RichFaces, MyFaces, IceFaces, PrimeFaces, SeamFaces.

Part of the problem is that there are several generation of "technology" that are mashed together in an overlapping mess. For example, J2EE6 uses CDI with Weld, but everyone already does this with Spring, which also does all sorts of other things so you might want to still use it even though you're doing DI with Weld in a J2EE container. Your objects might be being managed by some combination of JSF2 @ManagedBean lifecycle management, @EJB lifecycle management, Weld, or (if you're using it) Spring.

Or does JSF2 actually use CDI to manage its beans via Weld? Who knows! Every framework uses every other framework or has semi-transparent integration hooks for them. Every framework claims to make everything simple, while adding yet another layer of nightmarish complexity. Specifications inevitably have a reference implementation from Sun or JBoss, and an Apache implementation, so you have to figure out what the relationship between the implementation names and the spec name is for each, what they actually do, which implementation you might want to choose and why, etc.

When trying to learn about this stuff, it's hard to find materials that don't already assume you know the older technologies and just want to update to the newer ones. Hell, it's hard to figure out what the older ones and newer ones even are, or why you should want to use one over the other.

The J2EE platform releases are supposed to help with that, but seem to add complexity more than remove it, due to the need for backward compat, support for older frameworks and code that uses them, etc.

There's so much transparent management of application state, object lifetimes, request handling, etc that it becomes quite hard to figure out what your app is actually doing, how and when to do something, or how to alter some part of the automatic behaviour when you need to. As for debugging...

Java EE makes me understand how "non computer people" feel when listening to programmers talk.

On the other hand, it makes me feel better when really, really smart people point out that all this architecture makes it hard to see what you're actually trying to do.

... and all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.

.... and really, reading the manual makes it all much more approachable ;-)

Great service providers I use

2010-08-04T15:27:00.001+08:00

A few of the service providers I use deserve some credit for great work. They are:

Linode.com, a great VPS host with excellent prices, good service, and really, really good tools for VPS management and control. You'll never need to call support because you broke SSH or installed a busted kernel ever again, since you have a web-based virtual serial console and full bootloader control over the web interface. Check them out, they're excellent.
SimpleCDN.com, whose mirroring CDN service "Mirror Buckets" is truly incredible. I put about fifty bucks into it six months ago, and still haven't run out. It's about two thousand times cheaper than my old host's traffic charges - though admittedly that was an Australian VPS host with typically Australian bandwidth-prices-of-doom.
Internode, who are the only Australian ISP at time of writing who're offering native IPv6 to regular ADSL customers. They're otherwise pretty decent within the limits of a cut-price cut-throat industry. Imperfect, but they're in telecommunications, and compared to the telco/ISP average they don't even rank on the suck chart.

All hail GNU Parted

2010-08-02T12:31:00.001+08:00

A user bought in a dead laptop today. It was booting into Dell MediaDirect (an XP Embedded OS stored on the same disk as the main Windows XP install) instead of Windows XP, then bluescreening early in MediaDirect boot.

The user had accidentally hit the hardware button that boots the machine into MediaDirect instead of XP.

The machine booted fine under an Ubuntu live CD, and examination of the partition table quickly revealed that it was completely mangled. The primary partition that should've mapped Windows XP instead pointed to the same 2GB near the end of the disk that was mapped by the extended partition used by MediaDirect. So the partition table had two identical overlapping partitions, neither of which pointed to the real OS install.

The fix was trivial, thanks to GNU Parted.

$ sudo parted /dev/sda
rm
1
unit
%
rescue
0
100

In other words: Remove the primary partition (/dev/sda1) that should've been Windows XP, but instead pointed to MediaDirect (which was also on /dev/sda5). Then set units to percentage, and tell parted to scan the whole disk for lost file systems.

It'll find the original NTFS file system (for XP) and offer to add a partition for it. Easy.

You need to make sure the partition is flagged 'active' as XP's bootloader is a little dim. As I don't know parted well, I did this with:

$ sudo fdisk /dev/sda
a
1
w

... then rebooted into a working XP for the user.

The original cause is unknown, but perhaps this'll help others who've had MediaDirect mysteriously eat their partition table.

The Java needs language-level property support (and a reset)

2010-08-01T13:03:00.005+08:00

I find myself increasingly bewildered by some of the design choices in the Java world. The core language's use of accessor methods is a notable example.

Not that properties are the only bewildering Java design choice.

Since the very early Java days, this has been discouraged:

public class Something { 
  public int prop;
}

public class SomethingUser {
  public void outputThing(Something thing) {
    System.out.println("Thing " + thing + " has value " + thing.prop);
  }
}

in favour of an accessor-driven approach:


public class Something { 
  private int prop;
  
  public int getProp() {
    return prop;
  }

  public void setProp(int prop) {
     this.prop = prop;
  }
}

public class SomethingUser {
  public void outputThing(Something thing) {
    System.out.println("Thing " + thing + " has value " + thing.getProp());
  }
}

... which is verbose and annoying. On the other hand there are good arguments for why you want accessors - essentially they're a large part of the difference between using "structs and procedures" and using objects and messaging (OOP). Both can be good models, but if you're going to try for an OOP approach you need ways to trigger side effects when values change, or to return generated values as the internal representation of an object changes, both of which require accessors.

One might wonder, though, if something this important should be something the language can do for you in simple cases, so you only have to hand-write the accessors that do non-default things. Like the following (wishful thinking) pseudo-Java:

public class Something { 
  public property int prop;
}

public class SomethingUser {
  public void outputThing(Something thing) {
    System.out.println("Thing " + thing + " has value " + thing.getProp());
  }
}

This wishful thinking becomes more significant when the need to introduce property change listeners is considered. Java has a system for listening for changes to properties of objects and triggering events when these properties change. It's entirely at the library level, though, with no language support. A property change enabled version of our first example might look like this:

public class Something { 
  private static PropertyChangeSupport changeSupport = new PropertyChangeSupport(this);
  private int prop;
  
  public int getProp() {
    return prop;
  }

  public void setProp(int prop) {
     changeSupport.firePropertyChange("prop", this.prop, this.prop = prop);
  }

  public void addPropertyChangeListener(PropertyChangeListener listener) {
    changeSupport.addPropertyChangeListener(listener);
  }

  public void addPropertyChangeListener(String propName, PropertyChangeListener listener) {
    changeSupport.addPropertyChangeListener(propName, listener);
  }

  public void removePropertyChangeListener(PropertyChangeListener listener) {
    changeSupport.removePropertyChangeListener(listener);
  }

  public void removePropertyChangeListener(String propName, PropertyChangeListener listener) {
    changeSupport.removePropertyChangeListener(propName, listener);
  }
}

public class SomethingUser {
  public SomethingUser(Something thing) {
    thing.addPropertyChangeListener("prop", new PropertyChangeListener() {
      public propertyChange(PropertyChangeEvent evt) {
        outputThing((Thing)evt.getSource());
      }
    });
  }

  public void outputThing(Something thing) {
    System.out.println("Thing " + thing + " has value " + thing.getProp());
  }
}

Getting verbose, isn't it? Given how important the property change system is in Java, one might wonder why there's no language integration. Imagine a "bean" modifier that inserted property change support (we'd do this with a mixin class, but Java doesn't have multiple inheritance or mixins, so we need language support) and enabled property change notification on declared properties:

public bean class Something { 
  public property int prop;
}

public class SomethingUser {
  public SomethingUser(Something thing) {
    thing.addPropertyChangeListener("prop", new PropertyChangeListener() {
      public propertyChange(PropertyChangeEvent evt) {
        outputThing((Thing)evt.getSource());
      }
    });
  }

  public void outputThing(Something thing) {
    System.out.println("Thing " + thing + " has value " + thing.getProp());
  }
}

The property client side is still gruesome, but writing those property-change enabled beans is suddenly much, much less painful. You can remove the "property" modifier and implement the accessors yourself if you need to do anything interesting, but the 99% of boilerplate crap you have to write usually just goes away.

Rather better thought out proposals have been made by many others, such as this one, though like most of them the linked proposal fails to consider bean binding and change notification.

Unfortunately, rather than solving real-world pain points for Java developers, the Java language appears to be stuck in arguments over whether closures should be supported, endless nattering about reified vs erasing generics, etc. These are all useful, but there are so many pain points in the library that could be dealt with by a few language extensions.

Even a decent, standard, core way of doing mixins would help a lot. As things stand you have numerous aspect-oriented programming frameworks that half-solve the problem, but also tend to conflict with each other, add complexity, require annotation processors or other code generation, and generally be a pain to use.

You know Java is in a bad way when you start to miss coding C++.

(Yes, I know I can do this with Groovy. Sort-of. Of course, Groovy doesn't play well with some other Java features like anonymous inner classes, requires a whole new set of tools and tool support enhancements, its own dev env and compiler, etc. Why not fix the core language?)

Sanity in the J2EE / JSF world

2010-07-30T16:04:00.003+08:00

I was very impressed with this tutorial from CoreServlets.com, which explains JSF and JSF2 development without assuming that you know all the Java server-side acronyms and mess already.

It also has a rational, calm discussion of web framework selection, covering Google Web Toolkit (GWT), Wicket/Tapestry, JSF2, etc that focuses on what each is good for, rather than trying to name some One True Framework. In my reading on the J2EE world thus far, this is unique.

Another article that anyone looking to get into JSF2 should read is this DZone article summarizing how to handle some common tasks in JSF2. It won't make much sense until you've read some basics about JSF, like the tutorial linked to at the start of this post, but it's a great overview and refresher once you have.

Of course, as others have said very well indeed, getting into the Java Enterprise programming world of JSF2 etc may be a mistake.

A message for SCO OpenServer users

2010-07-25T12:57:00.001+08:00

I've seen a few queries on the PostgreSQL list from people who want to run PostgreSQL on SCO OpenServer, or upgrade ancient versions of Pg to modern ones on SCO OpenServer boxes. Some want to solve issues with connection count limits on their elderly SCO installs, so they can increase client counts above 100 or so. Sometimes people even want to upgrade to a new (ie post-1995, not truly new) SCO OpenServer release and want to know how Pg will cope.

I have a message for those folks, and the management behind them who're usually the ones pushing to stay on SCO.

Your boss may not realize that SCO basically dropped OpenServer as a product line in favour of UnixWare in the late 90s. Since then there was no significant work done on OpenServer. There's been no work done on it at all (as far as I can tell) since Caldera bought the SCO name and OpenServer product from the original Santa Cruz Operation, fired all the software engineers, hired some lawyers and sued world+dog. The Santa Cruz Operation renamed themselves Tarantella after their primary profitable product and went on with life, but "SCO" as a company is history.

OpenServer is dead, dead, dead. Any money put into products targeting openserver is a sunk cost, and you can't change that, but you should really avoid sinking more money into that mess. If your management is still sticking to OpenServer, they should probably read about escalation of commitment, a decision making tendency that's very dangerous and very easy to fall into if you don't think about it carefully.

Upgrading from 5.0.5 / 5.0.7 to 6.0 is like upgrading from Windows 95 to Windows ME in 2010. Or Mac OS 7.1 to Mac OS 9.2. You're upgrading from the corpse of an operating system to one that's still twitching feebly. This is not going to be a good way to invest time and money.

In case you think I'm just a Linux zealot flag-waving, I have a SCO OpenServer 5.0.5 box in the back room, running business critical applications. The apps are actually for Microsoft Xenix (yes, 1983 binaries) running in the Xenix persionality on OpenServer. I considered a port to OpenServer 6.0, but realized it was just slightly delaying the inevitable move to something modern.

So ... I keep it running - in VMWare*, since 5.0.5 runs about ten times faster as a VMWare guest on a Linux host than it does natively on the same hardware. It's faster because SCO doesn't use much RAM for disk cache, doesn't readahead, and is generally just sloooooow in its disk access and memory use strategies. The Linux guest in a vmware setup can cache the whole SCO OS and apps disk in RAM, making the whole setup much faster. It seems more stable under VMWare than running natively on modern hardware, too.

I'd recommend you do much what I've done. Move your SCO instances to VMs running under Linux. Provide modern PostgreSQL on the Linux host, and just compile libpq for the SCO guest. Then start work on migrating your app to run natively on Linux/BSD/Solaris/whatever.

* SCO OpenServer doesn't seem to run under KVM or qemu due to bugs and limitations in their SCSI emulation. VMWare Server is free, so just use that until you're free of SCO.

Java "Enterprise" tools: every problem half-solved with twice the required complexity

2010-07-22T12:04:00.013+08:00

The more I work with Java, the more I feel like it's a great core language surrounded by endless onion-like layers of increasingly awful frameworks and tools.

The Netbeans platform. JPA2. Hibernate / OpenJPA / EclipseLink / Swing. EJB3. OSGi. With tools like those, you'd think that a simple task like writing a business database application would be easy, right?

Ha! You spend so much time working around problems created by these tools' solutions to other problems that you're lucky to get any actual work done on your app.

Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly. -- Steve Yegge (2007, Codes Worst Enemy)

Examples demonstrating the use of JPA, Hibernate, etc are full of assumptions and limitations that render them useful only for utterly trivial toy projects. For tools that're touted as making it easy to build "scalable" apps, I find this interesting. One Hibernate+JSF tutorial, for example, writes:

One of JCatalog's assumptions is that no more than 500 products are in the catalog. All product information can fit into the user session.

We're building an Enterprise Application with n-teir Enterprise Architecture ... that's limited to many fewer items tracked than there are lines of code in the project. Does this seem off to you? If it's so easy to build scalable data access with Hibernate, why isn't it done in demoes and tutorials? My own experience suggests that's because it's not actually at all easy, it's painful and complex if possible at all.

This post expresses much of what I'm trying to say below, and expresses it much, much better than I can.

I should've just stuck to JDBC and core Java + Swing. Not that Swing is exactly nice, as its event-driven single-threaded design combined with rather less than event-driven models requires some complex thread-juggling to maintain a reliable, responsive app. Maybe just learned Ruby+Rails, PHP, or done it as a simple Python webapp. I'd be done in no time, with none of the stupidity involved in fighting the JPA to make it do what you need. My productivity in terms of real, usable application code produced has gone down to near-zero with these tools. There are endless demos on how great these tools all are (most of which demonstrate tiny toy apps with hopeless limitations) but there's no information out there on how to solve the problems with their real-world use and get a usable app going with them. See, eg:

Creating a CRUD netbeans application, which produces a completely useless toy app that fails to solve any of the interesting problems with desktop DB apps, like handling DB disconnects, 'net connection loss, latency, limited bandwidth, etc while maintaining a responsive and well behaved UI. I know, let's sweep those problems under the rug and pretend they don't exist! For just how complex some of these problems are, see this excellent article on using JPA with Swing.

For others considering combining these technologies: JPA is designed for the server-side world, where it lives in an application server with reliable low-latency access to the database and a pool of identical connections available to it. It claims to let you write database-transparent code that works with database entities as normal Java objects, largely taking care of the persistence of these objects for you. It offers transaction management features and is theoretically very nice to use.

On the server. Well, sometimes it's fine on the server, sometimes a horrifying train wreck of multi-layered duplicate code and copying that's in true Enterprise Application style.

Don't expect easy use of native database types, easy type mapping and translation (think: emulating boolean in crap databases that don't support it), though.

As for desktop and simple 2-teir apps: People will say JPA can work happily in desktop apps, but I suspect these people haven't written real desktop apps with it, or they're into over-abstraction, pain, and "Enterprise-y" design. You know, where you spend twice as long building the tools to build the tools to build the frameworks to control the enterprise architecture to model the modules that eventually contain some actual useful application-domain code as you do writing the code that does the work your application is for. Either that, or they think that having their app appear to crash/hang for long periods when the user's wifi drops, they go out of 3G cellular range, etc is OK. They also have no problem with transferring 1000x as much data is needed or making 100x as many database round trips as required so the app is incredibly, glacially slow on high latency connections.

Why is JPA so dreadful for desktop apps? For just one reason, see my previous post on issues with the interaction of JPA lazy fetching with the need to keep the EDT from blocking in Swing.

What it all comes down to is that your app's code really all does have to be aware of the database. It has to be aware that database access can be slow, can time out, and requires user feedback. It has to figure out what it'll need to know in advance and fetch that information in efficient batches. It has to selectively retrieve only what information it needs, and do so without having to call back to the database continuously to get information it missed. This is unavoidably hard to do, and trying to hide it from the app like JPA does makes it much harder, not easier, to get right.

Overall, my personal experience with JPA/Hibernate and friends like EclipseLink and TopLink Essentials has been that they're absolute productivity killers. They seem great at first brush, but the more you work with them the more you realise they're a clumsy solution and create more problems than they solve.

Hibernate and co can make deleting an object from a database into a complicated operation where you stare at inter-entity cascades until you're half blind.

Even the experts can't get it right. The NetBeans IDE generates rather broken JPA mappings by default, which can't even be used to remove an entity until they're fixed. Lest you think it's my UI code, I'm using an IDE-generated JSF2 interface too, and haven't typed a line of code in this project. Pity it doesn't work.

So: If you're going to use the NetBeans Platform or any Swing app, think very carefully about whether it's a good idea to use JPA, and understand the issues nobody likes to talk about. For that matter, think carefully about it if you're building a webapp too - you might well be better off with something like iBatis/MyBatis.

With Enterprise Java in general you'll get very used to this feeling:

Whenever I write code in Java I feel like I'm filling out endless forms in triplicate. -- Joe Marshall (aka jrm)

Quotes from here. Some of the later ones ring really, really true of "modern" Java development, where frameworks are stacked on frameworks to the point where you can't even see your own code anymore.

JPA (and Hibernate/EclipseLink/OpenJPA/etc) in desktop apps is awful

2010-07-22T10:59:00.003+08:00

Using the JPA API in desktop apps seems like a great idea. It provides a standard interface to Java ORM systems, letting you use annotations to map database entities to beans in a quick and convenient manner. Implementations may be plugged in as required, so you can switch from Hibernate to EclipseLink under the hood with little pain and fuss.

You'll find lots of examples and tutorials on the 'net that demonstrate how to use JPA to build simple "CRUD" apps with various frameworks, like the NetBeans platform, the (now defunct) Swing Application Framework, etc. They make it look like a quick, convenient and easy way to develop database access apps in Java.

There's just one wee problem... they don't work in the real world. With desktop apps, that is; I'm sure Hibernate and friends are great on the server in a 3-tier Enterprise Application with a suitable army of coders.

JPA 2.0 relies on lazy initialization of beans to be efficient. You don't want to download all the data you have on hand about 10,000 customers in your database just to display their names in a tree view. You want to retrieve just their names, lazily fetching most other attributes. Later, when the user goes to (say) edit the customer, you can just fetch the lazy attributes on first use. That's how JPA works, and it seems like a great approach that saves a lot of hassle.

Theoretically, anyway. JPA lazy fetching only works within the context of an open EntityManager session. Entity beans retrieved by an EntityManager typically have a much longer lifetime than that of the EntityManager, ie they become detatched from the database session. Unfortunately, detatched entities will fail to access lazily loaded properties, throwing an exception if access to a lazy property that's not yet loaded is attempted.

It might seem like the sensible thing to do is just keep the EntityManager around, so the entity beans remain attached to the session and can lazily load properties whenever required. Unfortunately this isn't very practical. For one thing, you'll often be using entity beans with APIs (like Swing models) that don't know they're entity beans and have no awareness of their connection to the database. Making this possible is half the point of JPA, but gives you no way to link the lifespan of the EntityManager to that of the beans it manages, except with the ugly hack of inserting a reference to the entity manager into each managed bean. Even then you can't guarantee to close the EntityManager when the last bean that uses it is being disposed of, because of the unreliability of on-finalize actions in Java. Letting the entity manager be quietly gc'd is unacceptable for the same reason - there's no guarantee it'll properly release its non-memory resources like any connection it's checked out of the pool, so if you don't explicitly close the entity manager you can leak connections from your pool.

Worse than that, though, is that even if you keep the entity manager around to handle lazy property loading and you've somehow solved the lifespan/scope issues, lazy loading can cause a thread to block waiting for database/network access whenever you access a bean property that happens to be mapped as lazy. You have no control over what thread the database access is done in, and no way to let a thread do other things while lazy properties are loaded. There's no way to respond to a call to"getCustomerName()" with "Er, try again later, I'm finding that out for you now". This is awful for GUI applications, because it means that the UI thread will block while the database is accessed without the ability to even display a busy cursor. What if the user's wifi drops, or they're on cellular? Your app just "crashed" as far as they're concerned.

Working around the blocking issue removes most of the benefits of JPA use. You have to do all your database work in a separate database worker thread (or set of threads). Remember Swing 101: don't block the EDT. Because you can't control blocking when lazy loading is used, you have to detach all entities before letting them escape the context of the database worker thread. This means that you either can't use lazy properties (so you waste a LOT of network bandwidth and loading time retrieving things you don't need) or your non-database components need to be aware of how to handle detatched entities with lazy properties.

Because there's no way to reliably tell what lazy properties of an object might be required when you're retrieving it, you'll find yourself reloading entities frequently as they're passed around the app. Each time there's a database access delay, and at each point you have to be able to handle the user going off and doing other things in the UI, or have to block the UI with appropriate feedback. Every GUI component starts to sprout code to check entities its passed to see if they have the required properties loaded and if not, return control to the EDT while reloading the entity with the required properties preloaded. For lazy loading to be useful in an environment where you can't block on database access, you need to know what properties of a bean will be used by what components in advance.

At this point, you may start to wonder if working directly with JDBC was such a bad thing after all. At least with JDBC it's easy to retrieve arbitrary subsets of your data, rather than fighting a framework that wants to have a fixed set of lazy/eager attributes and requires messing with "left join fetch" HQL hacks to selectively eagerly fetch.

Update: Someone else who's written about this has some pretty similar issues and no better answers. See http://blog.schauderhaft.de/2008/09/28/hibernate-sessions-in-two-tier-rich-client-applications/

I'm on the verge of recommending that people by Mac laptops

2010-07-20T12:10:00.001+08:00

.... solely because the amount of work required for Joe Average to take your average Windows laptop and strip the crapware out is too high.

I use Windows machines reasonably happily some of the time, and generally prefer them to Mac OS X for my own personal use (though I'll be on Linux whenever possible). However, to take your average machine from the state it arrives in to an acceptable state of usable, reliable service often requires:

A clean-install of the OS. This increasingly requires finding installation media, because OEMs don't ship Windows install DVDs anymore. Grr. If you can't do this, it's much harder to get rid of the mangled drivers and crapware apps from the machine.
Finding and installing *clean* drivers, not the butchered crap many OEMs ship. Frequently one can identify the chipset a part is based on and install the appropriate unmodified driver from Intel, RealTek, nVidia, etc, but sometimes this takes some wrangling. Video card and wireless drivers are notable offenders in this area.
Figuring out if any of the garbage the OEM shipped on the machine does anything useful, and installing it from the OEM's updates site (if you can find it). Vendors love to ignore standards on things like power management, building their own wacky interfaces that require their own weird drivers or control apps instead of using the standard platform interfaces like ACPI. Even if they do use the platform interfaces they often mis-implement them, then hack around the damage using replacement drivers in Windows. Grr.
Deciding on and installing antivirus software. Most users need it, despite the many downsides. One has to find an antivirus package that doesn't try to replace the perfectly good, reliable Windows firewall with its own crap, doesn't install buggy plugins into the browser, etc and just sticks to plain OS-level AV. Add the requirement that it not slow the machine to a glacial crawl and this can be challenging. I hate AV.
Educating the user. No, don't install multiple AV packages at once. Which updates are safe to install when prompted (and in fact vital) and some clues about how to tell the difference between them and scam popups from websites. Not that it'll do any good. Etc.

Windows, and the PC platform, is ugly. But so is Mac OS X and Apple's hardware, honestly. Microsoft's stuff is more backward compatible, has much safer and more reliable updates, and in my experience more stable on good hardware. Apple's stuff is shinier and can be nicer to use, has fewer (but far from no) UI warts, and currently has a lower (but non-zero) rate of malware, malicious apps, etc. OTOH, Apple's stuff is buggy, and much more likely to break with a system update. But Microsoft systems can't be used safely by a normal user without AV software, which is a nightmare.

They both suck. All computers suck. Argh!

How to make SCO OpenServer 5.0.5 printing work

2010-07-10T16:49:00.004+08:00

SCO OpenServer's printing is more than a little bit broken. lpsched tends to stop responding, and lpd seems to decide it doesn't feel like processing any jobs after a while. Killing lpsched and lpd then re-starting lpsched seems to solve the issue ... for a while.
I eventually got sick of it, and replaced the SCO print system entirely. I would've liked to use CUPS, but getting even the CUPS client to build on SCO was going to be an exciting effort in porting. The fact that one of the SCO patches broke the C compiler didn't help any, either.
In the end I wrote a simple Python script to relay print data to a Python server running on a modern Linux box, where the print data is handed to CUPS for printing. Since replacing the print system on the SCO box I've not had to intervene to fix SCO printing even once.
On the chance that others are stuck running this elderly operating system for their own scary legacy apps, I'm publishing the scripts here as a base for others to adapt to their own uses.

This print client program should be called instead of "lp" on your SCO box. It requires Python 1.5 from Skunkware to be installed. You should probably leave the original lp in place, instead adapting your PATH or your client apps to call this by preference.
The command line processing in this client program is incredibly simplistic. It expects to be called with a print spool file name, and optionally -dPRINTERNAME to specify a queue. It doesn't understand any other arguments. If you need more functionality, you'll have to extend the protocol and client.

https://github.com/ringerc/scrapcode/blob/master/scripts/sco_openserver505_replacement_lp/printserver.py

https://github.com/ringerc/scrapcode/blob/master/scripts/sco_openserver505_replacement_lp/lp.py

Brain-dead Bob

2010-06-29T15:57:00.007+08:00

iiNet's BoB (actually a Belkin F1PI243EGau) isn't very bright. For a highly capable device, "he" has some bad habits that just aren't being trained out.

UPDATE September 2010: Belkin/iiNet have finally released a firmware with a fix for the broken IPv6 lookup / DNS delay issue. See Firmware 1.2.37.86 in support answer id 2498. It's taken a year too long, but they got there in the end, and you don't have to ask for the Super Top Secret firmware from support anymore.

UPDATE2 October 2010: Belkin's firmware update fixed AAAA records, but SRV and TXT records are still broken. Morons. Same thing: TXT and SRV queries just time out. This is less than impressive QA, especially after being informed of the AAAA issue. One would think that Belkin would have a test suite to verify basic DNS support in their products that's run as part of the release process...

Original post continues:

Take DNS lookups. The built-in DNS forwarder doesn't understand AAAA lookups, the IPv6 address record query. This is important even if you don't use IPv6 because most modern browsers and operating systems issue IPv6 queries as well as, or before, IPv4 "A" queries when looking up hosts. They expect the IPv6 query to return promptly or fail promptly.

The BoB simply ignores such queries, causing the client DNS resolver to time out. This causes painfully long delays before the resolver realizes that the upstream DNS is never going to reply, and tries again with a regular "A" record lookup.

(Update 2010-07: there is a beta firmware ETH-WAN_v1.2.36.79 that fixes the AAAA record DNS issue described here, though A6 lookup is still busted. It's only available by calling iiNet support and asking for it, you won't find it on the website.)

I've observed this issue with Mac OS X and with modern Linux. I don't know if it affects Windows Vista and Windows 7 yet, but expect it does, though Windows might be smart enough to disable IPv6 lookup attempts after a few failed tries, figuring (correctly in this case) that the DNS server is a wee bit dim.

I reported this to iiNet months ago for another user, and it never got fixed. So I'm writing here to help people who have incredibly slow, glacial, painful DNS lookups - and thus web browsing, email, games, etc - on their iiNet BoB (Belkin F1PI243EGau) routers.

$ # Lookup via BoB DNS forwarder, the DHCP-issued default: 

$
$ time dig +short -t AAAA @10.1.1.1 google.com
;; connection timed out; no servers could be reached

real    0m15.009s
user    0m0.004s
sys    0m0.000s

$ # lookup via iinet DNS direct for same address immediately
$ # returns no result, just like it should.
$
$ time dig +short -t AAAA @203.0.178.191 google.com

real    0m0.028s
user    0m0.004s
sys    0m0.004s

Oh, it also defaults to sending a very "special" DNS search path over DHCP: "iiNet BoB". Because that's a valid domain name, and there's NO way that'll cause lots of bogus lookups for unqualified names...

Core 2 Duo CPU whine - workaround for Linux

2010-05-26T22:57:00.000+08:00

Do you use Linux on a laptop and fear you're going to be driven totally insane by the high-pitched whining noise (squeal/screech) coming from your Core 2 Duo CPU's badly designed cpu packaging and voltage regulation? Like proximity to your laptop gives you tinea?

Me too. A workaround that'll cost you a little battery life (but probably not more than a few tens of minutes out of four-plus hour runtimes) is to add processor.max_cstate=3 to your grub command line. This turns off the problematic C4 power state of deep CPU sleep at reduced voltage, but otherwise leaves power management unaffected.

( For Ubuntu users, dpkg-reconfigure grub2 will offer to let you edit the kernel command line. )

Using PKCS#12 client certificates with PgJDBC

2010-05-20T23:16:00.004+08:00

(This post is a reference copy of a mailing list post I made explaining how to use client certificates with PgJDBC if you wanted to be able to accept user-provided PKCS#12 files.)

Applications should use the core SunPKIX trust and certificate managers and the default SSLSocketFactory if at all possible, because they're flexible enough to handle pluggable authentication methods like hardware tokens with no app code changes. If you override the SSLSocketFactory and provide your own trust and key managers you lose a lot of the power of the system.

If, like mine, your app needs to offer a user a way to configure additional keys and trusted certs, just create your own JKS or JECKS KeyStores for the trust- and key- stores during startup (if they don't already exist), and set the javax.net.ssl. properties to point to them. The default Sun providers will then load those app-specific stores instead of the global cert- and key- stores.

If the user attempts a connection to a service that requires a client certificate, or a service whose certificate isn't trusted, you can catch the resulting exception, examine it to determine the root cause, and prompt the user to supply a client certificate or to install a trusted root cert as appropriate. Apache Commons Lang is useful here for inspection of deep exception chains.

If you'd prefer to load a user PKCS#12 file as the keystore its self - rather than importing it into a JECKS keystore - just specify its path in your app's javax.net.ssl. system properties via System.setProperty(...) before any SSL code is invoked. The default X509KeyStore will be happy to load and use it. Make sure to set javax.net.ssl.keyStoreType to pkcs12.

If you really do need to replace the X509TrustManager for some reason, use the certificate verification built-in to the JVM rather than re-implementing it. Create a Set<TrustAnchor> from your list of trusted X509Certificate instances from your trust store, and use it with CertPathValidator to validate a CertPath created from the X509Certificate[] provided to the TrustManager. This does proper PKIX validation a few lines of code.

Even better is to load the default "SunPKIX" X509TrustManager and X509KeyManager and put them before your own in the appropriate array when creating the SSL context in your SSLSocketFactory. The SSL context will try them in order. That way the user can still use all the pluggable features, but you can (eg) fall back to your X509TrustManager if the cert isn't trusted so you can display a "add to trust store?" prompt.

About the only downside with this that I've found is that there doesn't appear to be any way to force the SunPKIX code to re-load a KeyStore (the trust store or the key store) after SSL has been inited, so if the user adds a trusted cert or key they seem to need to re-start the app. You can get around it by using non-public com.sun api, but that's never a good idea.

If you're having problems in this area, you should read the JSSE reference guide, as it'll help you understand how it all works:

http://java.sun.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html

Using IET (ISCSI Enterprise Target) for Windows Server Backup

2010-05-05T12:31:00.003+08:00

Windows Server Backup in win2k8 server is fantastic - it's a consistent, snapshot-based automatic backup system capable of full disaster recovery / bare metal restore. I'm not a huge fan of much of the way the Windows servers work, but the backup setup is fantastic. With one wee flaw...

Manual backups may be made to a network share, or to a local volume then copied to a network share. Fuss free, but only with operator intervention.

Unfortunately, automatic scheduled backups require direct access to a drive, they won't work on a mounted NTFS volume or on a network share. This doesn't do me much good for disaster recovery, as even a USB2 or FireWire drive nearby has a good chance of being destroyed by anything that takes out my server. It rained (and hailed) in my server room last month, so I'm taking disaster recovery even more seriously, and a nearby HDD just isn't good enough.

I could run a FireWire 800 drive over cat5e to the near-site backup location, but that's surprisingly expensive to do, especially as I want redundant storage to protect against pesky HDD failures. I have a perfectly good Ethernet-connected Linux server with a 10TB RAID array running Bacula to back up everything else on thge network, and I'd prefer to just use it for Windows Server Backup too.

The solution: Win2k8 has a built-in iSCSI initiator. Simply turn the backup server into an iSCSI target, then use Windows 2008's built-in iSCSI initiator to connect to it so Windows Server Backup sees it as a local disk and can write backups to it. This turns out to be astonishingly easy, at least on an Ubuntu system.

Security notice

The following configuration does NOT authenticate the windows server to the iSCSI target via iSCSI mutual authentication, so it may be possible to trick the server into backing up onto a different server and "steal" the backup. It also passes the actual backup over the network in the clear, as it doesn't use IPSec. You may wish to address those limitations in your implementation.

It would be a very good idea to enable mutual authentication, but by time of writing I was unable to get it working. The win2k8 iSCSI initiator complained about secret length, even though the provided secret appeared to match its criteria and had been entered in the main part of the control panel where the mutual authentication secret is expected. Similarly, IPSec wouldn't be a bad idea to prevent your backups passing over the network in the clear.

Configuring the iSCSI target

First, install the ISCSI Enterprise Target software (IET):

apt-get install iscsitarget

Now provision a volume to export as a target. This may be a local raw disk or partition, a logical volume provided by LVM, or even a great honking file on one of your mounted file systems. I'm using LVM, so I'll just allocate a logical volume:

lvm lvcreate -n winimagebackup -L 300G backupvg

There is no need to format the volume; Windows does that. Just export it via iSCSI by adding a suitable target entry to /etc/ietd.conf (it might be /etc/iet/ietd.conf on your system):

 Target iqn.2010-01.localnet.backup:winimagebackup
        Lun 0 Path=/dev/backup/winimagebackup_iscsi,Type=blockio
        Alias winimagebackup
        IncomingUser iqn.1991-05.com.microsoft:winhostname xxxx

See the comments in the default ietd.conf and the ietd.conf man page for details on this. In brief:

Change "localnet.backup" to the reversed host and domain name of your target server's name (mine is called "backup.localnet").
Change "IncomingUser" to the user name you want the Win2k8 server to have to give to be permitted to connect, and "xxxx" to the password you wish to require. By default a 2k8 box will give the above user name, with "winhostname" replaced with the win2k8 box's hostname.
Set the path after "Path" to the location of your storage.
If you're using a file, you may need to specify "fileio" instead of "blockio" as the Type.

Restart ietd, and you're ready to connect the 2k8 box.

Connecting 2k8 to the iSCSI Target

Connecting to the target from win2k8 is similarly trivial. In the iSCSI Target control panel, in the "discovery" tab enter the dns name or ip of the target. Do not configure authentication (unless you've deviated from the ietd.confabove), just accept the dialog.

The server should appear in "target portals" and no error should be displayed. If successful, go to the "targets" tab, where you should see a target named "winimagebackup". Click "Log on..." to connect to it. Check the option to restore the connection at boot-time. Under Advanced, Configure CHAP authentication, using the password given in ietd.conf for IncomingUser under the target winimagebackup. Do not enable mutual authentication*. Accept the dialog, and the status of the volume should change to "connected".

Configuring Windows Server Backup

You're now ready to use Windows Server Backup with the volume. You do not need to format it under the disk mmc snapin before use. Just fire up Windows Server Backup and click "Backup Schedule", then follow the prompts, picking the iSCSI target as the backup storage when prompted.

Ubuntu Lucid and LTSP thin clients - wow

2010-05-03T13:23:00.001+08:00

For a while I've been suspecting that most people who do any Linux-based GUI develpoment neglect remote X11 and thin client considerations. Performance has been decreasing slowly and painfully, and more and more workaround have been needed to get systems to behave.

Well, no longer. The new Ubuntu release, lucid, with the latest gtk etc absolutely screams along. I'm seriously astonished that remote X11 can be this fast, given the round-trip-happy nature of the toolkits and the protocol's flaws. It's a real pleasure to use.

To everyone involved in gtk development - thank you!. Doubly so to those who looked into my bug reports on specific areas where gtk used excessive network round trips, particularly the Evolution compose window bug.

By the way, for those of you who deploy LTSP, another thing that'll make a huge difference to performance is to make sure your LTSP clients are on a private VLAN and then enable direct X11 communication between client and server with LDM_DIRECTX = Y in lts.conf. With this option you still use ssh to login and establish a session (so there's no godawful XDMCP to fight), but only the ssh login is tunneled and encrypted. During session setup, DISPLAY is redirected to point directly to the client's listening X server. This offers a huge performance boost, especially to slower/older clients without onboard hardware crypto engines. (Oddly, the 600MHz Via C3 boxes outperform the Intel Core 2 boxes when all X11 comms are encrypted).

Splash screens are a plague upon Linux systems

2010-04-30T11:09:00.001+08:00

How many splash screen tools can you remember appearing in at least one released version of at least one distro?

Too many, I bet, and all of them buggy.

In recent Ubuntu alone both usplash and plymouth have come to plague users. These tools purport to make bootup "friendly", but in fact:

Cover up important messages/warnings
Make recovering from issues during boot well-nigh impossible
Make boot-time fsck fragile and hard to interact with
Interact poorly with graphics drivers (xorg or kms)
Are much, much too hard to disable

Unfortunately, they don't even have a standard mechanism for disabling them. "nosplash" on the kernel command line used to work most of the time, but Plymouth displays a splash screen if the sequence "splash" appears anywhere in the kernel command line - including mid-word. It'll merrily accept "nosplash" as a request for a splash screen. With plymouth you must instead omit "splash" from the kernel command line entirely - and woe betide you if something else you need happens to include those characters!

Even better, Plymouth can't be uninstalled without ripping the guts out of an Ubuntu lucid system completely. It's wired deep into the package dependency system.

Argh. I'm coming to dread distro upgrades because I have to learn how to get rid of the blasted splash screen all over again. If only they'd stay uninstalled...

Intel, where's my EFI Network/USB/FireWire Target Disk Mode?

2010-04-24T14:37:00.000+08:00

I'm torn.

I really don't like working with Mac OS X (or Windows, for that matter - I'm a Linux user by preference) .... but I love some features of Apple's hardware. The build quality and disk bays of the Mac Pro, for example. But above all else, what I love and envy about macs is ... Target Disk Mode. It's a service tech and sysadmin's dream.

Intel likes to make a lot of fuss about all its fancy in-chipset managment features, yet it seems to lack that one most crucial and handy feature - a Target Disk Mode equivalent. C'mon Intel, you can do better than this! You can not only implement FireWire target disk, but Ethernet-based iSCSI Target Disk for true sysadmin heaven. For bonus points, add TPM support so only authorized service techs for the company can get in, and use the built-in network management features to let admins remote-reboot a machine into target mode.

Sadly, I suspect the reason we're not all using this is that the "good" (cough, cough) old PC BIOS is still malingering, and failing to decently give way to EFI/OpenFirmware/whatever like it should.

Use Linux software RAID? Schedule periodic RAID scrubbing or lose your data

2010-04-23T15:40:00.003+08:00

This post is bought to you by the fun of unnecessary wasted time and work rebuilding a server after a double-disk RAID array failure. RAID scrubbing is essential - and is supported by Linux's software RAID, but not used without explicit user action.

Linux's `md' software RAID isn't unique in this, but as its use is so wide spread it's worth singling out. No matter what RAID card/driver/system you use, you need to do raid scrubbing. Sometimes RAID vendors call this a "patrol read", "verify" or "consistency check", but it's all generally the same thing.

Why you should scrub your RAID arrays regularly

I had a disk fail in the backup server (some time ago now). No hassle - replace it, trigger a rebuild, and off I go. Unfortunately, during the rebuild another disk was flagged as faulty, rendering the array useless as it had a half-rebuilt spare and a second failed drive.

You'd think the chances of this were pretty low, but the trouble is that the second failed drive will have developed just a couple of defective sectors (a SMART check confirms this) that weren't detected because those sectors weren't being read. Until the drive was sequentially read during the rebuild, that is.

To reduce the chance of this, you can periodically verify your arrays and if bad sectors are discovered, attempt to force them to be remapped (by rewriting them from redundant data) or failing that fail the drive. This will also detect any areas where different disks disagree on what the correct data is, helping you to catch corruption caused by failing hardware early.

Unfortunately, Linux's software RAID doesn't do this automatically.

A simple shell script like this, dropped in /etc/cron.weekly and flagged executable, will save you a LOT of hassle:

#!/bin/bash
for f in /sys/block/md? ; do 
    echo check > $f/md/sync_action
done

Make sure to TEST YOUR EMAIL NOTIFICATION from mdadm, too. If a drive fails and you never get notified, you're very likely to lose that data the moment anything else goes wrong.

Use S.M.A.R.T too

For extra protection from failure, install smartmontools and configure smartd to run regular "long" tests of all your RAID member disks, so you're more likely to discover failing disks early.

Unfortunately, many consumer-oriented disk firmwares lie to the host and try to cover up bad sectors and read errors - probably to reduce warranty costs. Manufacturer's disk tools tend to do the same thing. Some even seem to lie during S.M.A.R.T self-testing, re-allocating sectors as they find bad ones and then claiming that everything is fine. In fact, I've actually had a consumer SATA drive that can't even read sector 0 return PASSED when queried for a SMART general health check, though at least it failed an explicitly requested self-test.

My point is that SMART testing alone isn't sufficient to ensure your disks are trustworthy, you really need to use a redundant array with some kind of parity or data duplication and do proper RAID scrubbing. And, of course, good backups.

If you use "RAID friendly" disks (usually the same physical drive with honest firmware and a much bigger price tag) you shouldn't have as many issues with SMART self-tests.

Battery backed cache for Linux software raid (md / mdadm)?

2010-04-23T14:49:00.004+08:00

Linux's software RAID implementation is absolutely wonderful. Sufficiently so that I no longer use hardware RAID controllers unless I need write caching for database workloads, in which case a battery backed cache is a necessity. I'm extremely thankful to those involved in its creation and maintenance.

Alas, when I do need write-through mode (write caching), I can't use mdadm software RAID. There's actually no technical reason hardware already on the market (like "RAM Drives") can't be used as write cache, it's just that the Linux `md' layer doesn't know how to do it.

I say "it's just that" in the same way that I "just" don't know how to fly a high performance fighter jet with my eyes closed. Being possible doesn't make it easy to implement or practical to implement in a safe, reliable and robust way.

This would be a really interesting project to tackle to bring software RAID truly on par with hardware RAID, but I can't help wondering if there's a reason nobody's already done it.

Are you wondering what write caching is, why write caching is so important for databases, or how on earth you can safely write-cache with software RAID? Read on...

WHAT IS WRITE CACHING?

Write caching (often called "write back mode") refers to a storage device that copies data written to it by the host OS into some kind of fast non-disk storage and then tells the operating system that data has hit the disk before it truly has. Not only does this provide incredibly fast fsync()s, but it also lets the storage device intelligently bunch up many small writes into fewer bigger writes to nearby disk regions, massively speeding up random write speeds. If your workload is mostly random writes with frequent fsync()s then you can expect speedups of tens or hundreds of times when enabling write caching.

For write caching to be safe, the fast storage used for cache has to be persistent even in the face of system power loss, sudden OS reset, removal of the RAID card/hard drive from the computer, etc. If it is not, any sort of failure will leave your data in a horrifying half-written half-lost state made even worse by the out-of-order writes done to improve random write speeds. It doesn't bear thinking about.

So - all it takes to make write caching perfectly safe is robust, persistent storage for the cache, and with such storage you can achieve orders of magnitude improvements in performance. Sound interesting?

WHY USE WRITE CACHING FOR DATABASES?

Database workloads benefit from write caching because they tend to require strict guarantees about what has hit disk. They also tend to do a lot of writes across random and widely scattered parts of the storage. Because of their need to ensure ordering of their writes, they tend to force those random writes to disk in order, preventing the RAID controller from accumulating them and combining them into bigger more ordered writes.

Given how bad rotational storage is at random writes, this is pretty much a worst-case workload for storage performance.

Even if your particular workload doesn't care about a few lost transactions so it doesn't require the database to promise that data has hit disk before returning from COMMIT, the database its self tends to require strictly ordered writes to avoid severe corruption of its storage in the case of unexpected power loss or reboot. This is particularly true for database systems like PostgreSQL that do write-ahead logging, a crash-integrity strategy somewhat like file system data journaling.

WHY DO YOU NEED A HARDWARE RAID CONTROLLER FOR WRITE CACHING?

Pure software RAID working only hard disks cannot provide write caching. It can cache writes in RAM, but if the OS suddenly crashes, the user hard-resets the machine, or the power goes, your data is in a truly messy half-written, half-lost state. The software raid system simply has no suitable storage for the write cache.

Hardware RAID controllers generally use ordinary DDR/DDR2 DIMMs (standard PC memory) plugged into the controller as their cache. Because this memory is erased when it loses power, they also include a small battery on the RAID card that maintains power to the memory for many hours, giving you time to get power back to the system before the cache contents are lost and your data is corrupted. When power is restored, the RAID controller resumes where it left off, writing any data it finds in the battery-backed cache memory.

There is nothing that inherently prevents software RAID from using the same strategy. All it needs is somewhere persistent to put the cache. There's an obvious way to provide such cache. Exactly the same option used by hardware RAID controllers can be used for software RAID - a DIMM or two on a PCI Express card or even a SATA "RAM drive" would do, so long as it had a battery to keep that memory alive across power outages.

So, in fact, you don't need a hardware RAID controller for write caching at all, you just need somewhere to put the cache.

IF SOFTWARE RAID CAN DO WRITE CACHING WITH SUITABLE CACHE MODULES, WHERE ARE THOSE MODULES?

Given the huge costs of true hardware RAID controllers, especially with the large premium charged by most vendors for an add-on battery backup unit, you'd expect that there'd be plenty of options out there, and that software RAID implementations would take advantage of them.

Yet, oddly, the only software RAID implementations I can find that support write-through (caching) mode with a battery-backed cache are vendor-specific "host raid" implementations embedded in drivers for "fakeraid" cards. These cards pretend to be hardware RAID, but really do the work in a driver in the operating system. The hardware is just a plain old SATA or SAS controller with an on-board BIOS Option ROM that understands enough of the RAID layout that it can read the boot loader and get the OS up and running to the point where the drivers load. It's easy for the vendors of these cards to add BBU / write cache support, since the software RAID is tightly bound to particular hardware that they can just add a RAM slot and a battery to. Unfortunately, they tend to add a rather impressive price tag as well.

I find myself wondering where the "write-through mode cards" for the built-in software RAID features in Linux, Mac OS X and Windows are. The success of fakeraid cards with BBU shows there's a market, so where are the devices?

Well, they're right here . "RAM drives" are pretty much perfect for the job. They're fast, battery backed storage that's just what's needed. It'd be nice if they offered direct PCIe access to the memory via an onboard ACHI-compatible host interface (or custom driver) rather than having a real SATA interface, but that's far from vital.

Unfortunately, the RAID implementations in Windows, Mac OS X and Linux don't know how to use persistent cache like this to enable write-through mode! It's out there, cheap, and ready for the taking, but the software just doesn't support it. md/mdadm support storing a write-intent bitmap for faster crash recovery, but it's just not close to the same.

Wouldn't it be awfully handy if md/mdadm supported an external cache module? Software RAID at hardware-RAID speeds.

An aside: WHY NOT USE AN SSD FOR WRITE CACHE STORAGE?

"Why not use an SSD?" you ask? Good question. Their random write performance is awful, but if you use them as a cyclic write buffer they're wonderful, and they'd be a near-perfect choice as they don't require backup batteries. Unfortunately, they're generally incredibly bad at doing lots of small writes, even if those writes are sequential, because of their erase blocks. Only high-end SSDs with internal cache and a big capacitor to let them finish writing after power-loss would be suitable, and they're way more expensive than a board with some DIMMS and a battery would be. You're better off with hardware RAID cards or a dedicated cache module for software RAID.

Before you say "Intel's X-25 series address the issues with small writes" ... try getting one of them, doing a bunch of writes, and yanking the power. See if what you see when you restore the power makes any sense at all. Those drives apparently don't have the juice to finish writing their internal cache out when they lose power - they're very much like software RAID in write-through mode without a BBU - ie incredibly unsafe. I've heard horror stories about them on the PostgreSQL mailing list, and wouldn't use them except on a machine with a really good UPS. All Intel would need to add would be a great honking capacitor or an itsy little backup battery to give it enough juice to do an emergency write-out, but presumably for cost reasons the devices don't offer anything like that.

RELATED LINKS

very experimental write-back cache support patches that only use unsafe, non-BBU main memory as cache: http://lwn.net/Articles/229976/

LVM vs VSS - it's no contest

2010-04-16T15:03:00.001+08:00

Richard Jones writes about Microsoft's Virtual Shadow Copy Service (VSS, not to be confused with Visual Source Safe), and laments the lack of any equivalent or good alternative on Linux servers.

I couldn't agree more, but there's more to it than what he focuses on. Application consistency is in fact the least of the problems.

APP CONSISTENCY

App consistency and pausing is missing in Linux, but is in principle not hard to introduce. It is something where D-BUS can be very useful, as it provides another piece of the puzzle in the form of inter-application and application<->system signaling. The trouble is that, like with most things in the Linux-related world, some kind of agreement needs to be reached by interested parties.

All it really takes for app consistency is two d-bus events:

"Prepare for snapshot - pause activity, fsync, etc."
"Snapshot taken/failed/cancelled, safe to resume"

... though there needs to be some aliveness-checking and time-limiting in place so that a broken application can't cause indefinite outages in other apps by not responding to a snapshot message.

Off the top of my head, products that'd benefit from this include Cyrus IMAPd, PostgreSQL, MySQL, any other database you care to name...

... IS ONLY A SMALL PART OF THE PUZZLE

As compared to VSS, using dm-snapshot (LVM snapshots) on Linux suffers from a number of significant deficiencies and problems:

Snapshots require you to use LVM. LVM doesn't support write barriers, so file systems must use much slower full flushes instead.
Snapshots are block-level not file-system-level, so the file system isn't aware of the snapshot being taken.
Because snapshots are block-level and not filesystem-aware, the snapshot must track even low-level file system activity like file defragmentation, erasing free space, etc. This means they grow fast and have a higher impact on system I/O load.
Accessing a snapshot requires mounting it manually to another location and faffing around with the changed paths. There's no way for an app to simply request that it sees a consistent point-in-time view of the FS instead of the current state, as in VSS. This is clumsy and painful especially for backups and the like.
The file system being mounted has to be able to cope with being mounted read-only in crashed state - it can't do journal replay/recovery etc. LVM doesn't even give the *filesystem* a chance to make its self consistent before the snapshot. Some file systems, like XFS, offer manual pause/resume commands that may be issued before and after a snapshot is taken to work around this.
Snapshots don't age out particularly gracefully. You have to guess how much space you'll need to store changes, as LVM isn't smart enough to just use free space in the VG until it runs out. The snapshot's change tracking space is reserved and can't be used for anything else (even another snapshot) until the snapshot is freed. If you over-estimate you can have fewer snapshots and need to keep more free space around. If you under-estimate your snapshot may die before you finish using it. Argh!

So: even with the ability to pause app activity, there's unfortunately a lot more to be done at the LVM level before anything even half as good as VSS is possible. LVM has been progressing little, if at all, for quite a few years now and some of these problems (such as the namespace issues and lack of FS integration) aren't even practical to solve with a block-driver level approach like LVM's.

At this point, one can only hope that BTRFS can do a better job, so that we can switch to runnning btrfs on mdadm raid volumes and breathe a sigh of relief.

The Australian Maginot Line - because it worked great last time

2010-04-16T15:03:00.000+08:00

Peter Thrush of ICANN recently commented that the Australian Internet Filter proposal is akin to the Maginot Line of WWII French fame. We all know how well that worked.

This is a surprisingly good analogy. The Maginot line presumed that the attacker would do what was expected of them, and wouldn't take the defenses into consideration when planning what they were doing. In much the same way, the Australian internet filter presumes that if it blocks what people do now, they won't change their behavior to circumvent the blocking with trivially available tools and techniques like encryption, tunneling, outside proxies, etc.

We already know that's an invalid assumption - not only is it rather contrary to general human nature, but it's being seen over and over in China with the Great Firewall. This despite the fact that China's Great Firewall is much more restrictive than Australia's is ever likely to be even under the most moralistic, conservative, idiotic government. Let's not forget, also, that in China it can be unhealthy to circumvent blocks that prevent you from accessing or posting information that's not meant to get around ... something I don't see becoming the case here.

So - in much more hostile circumstances, people still just waltz through the Great Firewall. Heck, I've done it myself - I had a workmate in China who needed unfiltered access, and it was the work of a few seconds to help him set up an encrypted SSH tunnel to a proxy on work's servers from which he could get to whatever websites he liked and do so undetectably. It's not even possible to tell that the encrypted data is web browsing data rather than something else.

Once again, it's clear that the only way the internet filter can work is if it's a whitelist. If a site isn't approved, you can't access it. If a protocol can't be inspected and content-filtered, it's blocked. No encryption of any sort may be used. Even that's imperfect due to cracking of whitelisted sites and use of them for proxies, etc.

It's a dumb idea. Why are we still wasting time and taxpayer money on such blithering idiocy?

Scripts to add replaygain tags to a flac collection

2010-04-16T15:01:00.001+08:00

If, like me, you like to be able to play your music without constantly having to lunge for the volume control, and you store the master copy of your (ripped) CD collection in FLAC format you might be interested in a quick way to (re)calculate all the ReplyGain tags on your music. I'll also go into how to automatically create smaller MP3 copies of the files with the replaygain bias "burned in" so stupid players still get the volume right.
Finally, there's another little script here that can be used to fix up issues like inconsistent artist naming. It only does "The X" -> "X, The" at the moment, but it's easy enough to extend.
Read on if you're happy with shell scripting (and a Linux or advanced Mac OS X user).

The following shell snippet will scan the target directory, in this case $HOME/flac, and calculate ReplayGain for all .flac files within it. It assumes that each directory containing any flac files may be treated as an album for purposes of ReplayGain tagging. None of the other track metadata is touched and the audio stream is not altered in any way.
You'll need the flac tools installed since metaflac is what we'll use to calculate and apply replaygain tags. On an Ubuntu system the flac tools are in the (surprise!) flac package.
Copy this to gain.sh:

#!/bin/bash
if test $# -lt 1 ; then
        echo "Usage: $0 target-dir [target-dir [target-dir [ ... ] ] ]"
        exit 1
fi
find "$@" -type d -execdir \
 bash -c 'echo Scanning `pwd`/{}; if ls "{}"/*.flac >&/dev/null ; then metaflac --add-replay-gain "{}"/*.flac; fi' \
 \;

Then invoke as sh gain.sh $HOME/flac (assuming $HOME/flac is the directory you want to recursively retag). You may specify more than one target directory to scan.
Once you've tagged your flac files with track and album level ReplayGain information you can play music from different publishers, in different styles, and of different ages without some of it being almost inaudible and other tracks mind blowingly loud. Your player must support ReplayGain, of course.
Rhythmbox (if you use it) doesn't apply ReplayGain by default, but does support it. To enable:

gconftool-2 --type bool --set /apps/rhythmbox/use_replaygain 1

No idea about iTunes and friends - can they even play flac?
Handily, you can also bulk convert your flac formatted collection to other formats (say, mid-quality MP3 or vorbis) for something like a portable player or a laptop with a small disk and in the process "burn in" the volume bias indicated by the ReplayGain tagging. That way players that don't understand per-track preamp, replaygain, etc will still produce reasonable volume levels without you having to re-encode your tracks from a lossy source and get the usual horrid results.
The lame mp3 encoder automatically calculates replaygain, but it doesn't handle per-album gain. You can instruct it not to apply replaygain and instead apply the preamp recorded in the flac metadata with the undocumented flac decoder option --apply-replaygain-which-is-not-lossless. There may be better ways; ideally you could specify an encoding preamp to the mp3 encoder.
This script will scan the directory flac in the current directory, and write mp3 files to the mp3 directory under the current directory. Output will be medium/high quality VBR MP3 with gain burned in to the audio stream and proper ID3 tags included. You can run one instance of the script per CPU core for best performance, as the instances will communicate and make sure not to step on each others' toes or duplicate work. You will need the lame MP3 encoder installed to use this script. The script will skip files that already exist in the target directory, so you can re-run it after ripping your latest CD acquisitions and it'll only encode the new files.
Of course, only a crazy person would run this without first making backups of their music collection. Then again, you'd have backups already, right? Right? Sigh.
Save to flac2mp3.sh and chmod a+x flac2mp3.sh:

#!/bin/bash

if test "$#" -lt 1 ; then
        echo "Usage: $0 target-directory"
        exit 1
fi

function gettag() {
        awk -F = /^${1}=/' { print $2; }' /tmp/tags-$$
}

function rmlock() {
        trap '' EXIT
        rm -f "${lockfn}"
        exit 2
}

if test "$1" = "convert" ; then
        # Called by self to convert a track. $2 is source track name,
        # $3 is source dir prefix, and $4 is target dir prefix.
        #
        # The track metadata is extracted, then the track is decoded with replaygain
        # bias applied during decoding. High quality VBR MP3 encoding is then done
        # on the adjusted audio stream. The end result is a decent quality MP3
        # that will play at a sane volume even on players that don't understand
        # ReplayGain MP3 headers/tags.
        ifn="$2"
        ofn="${2/.flac/.mp3}"
        ofn="${ofn/$3/$4}"
        if ! test -a "${ofn}" || test "${ifn}" -nt "${ofn}" ; then
                # The file is either absent in the target directory or is older
                # then the source file.
                #
                # Check to see if another instance of the script is currently encoding
                # this file, and if not, start encoding it ourselves. There's a race here,
                # in that another instance may start work on the file between when we check
                # and when we start work, but we don't actually care as the worst outcome
                # is a little wasted CPU time.
                #
                # We'll use a lock file in the target directory to indicate activity.
                #
                odir="`dirname "${ofn}"`"
                lockfn="${odir}/.`basename "${ofn}"`.lock"
                if test -e "${lockfn}" 2>/dev/null ; then
                        # Appears locked by another instance. Is it still alive?        
                        if kill -0 $(cat "${lockfn}"); then
                                echo "Skipping ${ifn} - locked by another instance"
                                exit 1
                        else
                                echo "Clearing stale lock file for ${ofn}"
                                rm "${lockfn}"
                        fi
                fi
                # Notify any concurrent instances that we're working on this file
                trap rmlock EXIT SIGTERM SIGINT SIGQUIT SIGHUP
                mkdir -p "${odir}"
                touch "${lockfn}"
                # then process it
                echo "$$" > "${lockfn}"
                metaflac --export-tags-to=/tmp/tags-$$ "${ifn}"
                echo -n "Converting: ${ifn} ..."
                flac -d "$2" -c -s --apply-replaygain-which-is-not-lossless 2>/dev/null \
                        | nice lame -S -h -V 3 --noreplaygain \
                                --tt "$(gettag TITLE)" --ta "$(gettag ARTIST)" --tl "$(gettag ALBUM)" \
                                --ty "$(gettag DATE)" --tn "$(gettag TRACKNUMBER)" --tg "$(gettag GENRE)" \
                                --ignore-tag-errors \
                                - "/tmp/working-$$.mp3"
                if [ $? ] ; then
                        mv /tmp/working-$$.mp3 "${ofn}"
                fi
                echo " done"
        else
                echo "Skipping ${ifn} - destination exists and is up to date"
        fi
        exit 0
fi

# For each flac file in the source tree test to see if an MP3 must
# be created and if so, make one.
find "$1" -name \*.flac \
        -exec "$0" convert "{}" flac mp3 \;

Here's another script that goes through the collection and changes FLAC metadata to rename, eg, "The Cure" to "Cure, The". It's easily adapted to do other things too.

#!/bin/bash
#
# Fix some common metadata issues on my flac collection, including rewriting
# artist names from "The X" to "X, The".
#
# Typical usage:
#    find flac -type f -name \*.flac -exec bin/cleanup_flac "{}" \;

if test $# -lt 1 ; then
        echo "Usage: $0 target-file.flac"
        exit 1
fi

function gettag() {
        awk -F = /^${1}=/' { print $2; }' /tmp/tags-$$
}

metaflac --export-tags-to=/tmp/tags-$$ "$1"
artist="$(gettag ARTIST)"

# Convert "The X" -> "X, The"
echo -n "Testing \"$1\": "
if test "${artist:0:4}" = "The "; then
        artist="${artist:4}, The"
        metaflac --remove-tag="ARTIST" --set-tag="ARTIST=${artist}" "$1"
        echo "artist changed to ${artist}"
else
        echo "ok"
fi

The Great Australian Firewall just won't work

2010-04-16T14:57:00.003+08:00

The proposed Australian Internet censorship rules will not work. Like most such blacklist-based schemes without active human monitoring it can be trivially bypassed by anybody capable of following simple instructions. As such, all the people it's designed to stop (like kiddie porn scum) will ignore its supposed blocking effects completely. Meanwhile we'll all have to live with the performance degradation, reliability problems, and so on.

It can't work because:

Blacklists are not useful if you don't know what to block. Private websites (content unknown until you have the password), websites that switch content dynamically based on source IP or authentication details, etc all defeat blacklists unless an insider reports the site as a problem.
Blacklists are always out of date. A content provider can hop domains and IP addresses much faster than a blacklist has any hope of keeping up with, and can easily inform interested parties of the changes over IRC, email, etc.
HTTPS prevents content inspection. A filter cannot content-filter encrypted communication like HTTPs, so all they can go by is the IP address and reverse DNS. If the blacklist doesn't cover that IP address, the communication must be permitted.
Encrypted tunnels and VPNs can completely bypass the filtering. You can't block all encrypted traffic through the filter. You'd prevent secure online payment, remote access to business networks by teleworkers, system administrators logging in to maintain their networks, etc. To bypass the filter all somebody needs to do is have a host outside Australia that runs a proxy server and accepts SSH connections, terminates PPTP VPNs, accepts IPSec connections, or any number of other encryption options. The filter cannot see the content of the data stream as it's encrypted, but the user has free and unimpeded web browsing through the proxy. You can get a web host with SSH access for around $50/month, and groups of people could easily get together to set up proxy hubs. In fact, they're certain to, and few of them will be criminals - most will just want their web browsing to work.
Peer-to-peer file transfer clients support encrypted communication. Even if there were effective methods to fingerprint files (such as images) transferred via BitTorrent and such, those methods are easily foiled. A client may encrypt the data so that only the sender and recipient know what it is, rendering a filter between the two unable to tell the difference between "My Gradmother's 100th Birthday Videos" and kiddie porn. Even if the P2P client didn't support encryption, transferring an encrypted zip file with the password or decryption key in a text file would work just as well.
Files can be transferred out-of-order and cannot be fingerprinted unless complete. This means that you either need to block out-of-order transfers (which will stop all sorts of legit things that programs you use every day, like Adobe Reader, rely on) or have the filtering system completely download the file before passing any of it on to the recipient. The former isn't acceptable if you claim the impact of the filter will be minimal on legit users. The latter is a trivial denial of service - request 1kb of every multi-megabyte PDF you can find, forcing the filter to download all of them without costing you any significant local resources. Yet this is a perfectly legitimate thing to do, and the filter should not prevent it. A store and forward system like this would also massively slow all web browsing, and require mind boggling amounts of fast storage to successfully operate.

None of the above is difficult. You might not know the technical terms involved, but actually using it is absolutely trivial - along the lines of "visit the website https://10.101.122.144/", "Go to https://safeandhappy.com/secretlogin.php", or "Click on connect to VPN and enter the address 10.101.122.144 with username fred and password fred".
So, the filtering won't stop any criminals except the very, very stupid. For example, it's been proved time and time again that child porn traffickers, who're one of the groups being bought up repeatedly in arguments in favour of this filter, are quite sophisticated in their use of technology to hide their activities. They use encryption and private secure web sites heavily, for one thing. As a result, they may not even notice the introduction of filtering. The only ones who will get caught or blocked are the occasional casual sickos searching the Internet. While disgusting, they are not the dangerous ones, because they're not the ones abusing children to supply their habit.
The same applies to basically any illegal activity. Stop people obtaining "terrorist training materials"? Yes, you'll stop legitimate researchers afraid to bypass the filters, the incredibly stupid, and nobody else.
As if its total ineffectiveness wasn't bad enough, such filtering schemes introduce a huge and complex system that sits between Australia and the rest of the 'net. There will be an inevitable reliability impact due to outages of the system. Additionally, it'll block content that should not be blocked - even the best systems have false positive rates as high as a couple of percent, or 1 in 50 files/pages. I don't even want to imagine the effect that'll have on automatic software updates, XML-RPC, and other non-web-browsing communication that often happens over HTTP. If they try to filter traffic on ports other than 80, it's going to be vastly worse in terms of the amount of software it just breaks.
The filter is clearly nonsensical from a practical and technical perspective. That's without even getting into the argument about whether it is right.
If there was a system that could effectively catch (blocking is pointless; they'll just use other methods) people trafficking child porn and the like, I'd be all for it - even if it had some significant side effects. There isn't, though, and no technical advance can create one so long as you consider encrypted communication without a government decryption backdoor to be allowable.

Whitelists

Fundamentally, the only sort of filter that actually works is one that disallows all communication of any sort not known to be "safe". No encrypted communication may be permitted except to sites known to be entirely legal, so most secure logins and online payment is out. All websites must be checked in depth before being added to the whitelist ... and must be regularly re-checked to make sure they're still compliant. No user-generated content websites can be allowed unless they check all submissions before posting.
Even then, it's necessary to have a huge team of humans working for the filtering provider check random samplings of URLs being requested and view random samplings of images, etc. An apparently legit site can trivially hide a back door into a secure private area where less nice things go on. The site operator might not even know the back door is there, since servers are regularly broken into and defaced or used to host malware by malicious 3rd parties.
It's also necessary to fingerprint files and compare them to a list of known prohibited files as another step toward detecting this sort of thing. This requires store-and-forward proxying (since you can't block a file you've already passed on to the recipient), which as noted above is insanely expensive in storage and processing power.
Despite all that work, malicious parties can still get a lot done between breaking in to a server and the break-in being detected. File fingerprinting is of limited use as it's not particularly reliable and any system without absurdly high false positive rates is easily tricked by small changes in the file.
It's also possible to hide information in images and certain other file types in such a way as that it's actually undetectable but can still be extracted if you know it is there. This is called steganography. The amount of information that can be hidden in a given image is fairly small, but over time quite a bit can be transferred. This technique can be used to undetectably transfer whatever you want in blog post after blog post of picures of cats.
The result of a whitelist scheme (which is not proposed, but is the only thing that could work) would be that only a tiny percentage of the Web would be accessible. There could would be no email hosted outside Australia. No peer to peer file transfers. No access to Facebook, LiveJournal, etc as untrusted user content can be put up there. No instant messaging until/unless the companies involved worked with the filter providers to enable filtering of messages and file transfers done through their programs. No way to administrate servers located outside Australia (no SSH). No way for remote workers to log in to company networks internationally. The list is almost endless, actually ... and even with all these consequences it'd just make life a bit more difficult for the people you're trying to stop.
Note that you can allow encryption if you force all encryption users to disclose their decryption keys to the government. However, the computational overhead of testing and decrypting each data stream would be astounding. Furthermore, in some cases you can't tell if you have the right key or not until the whole data stream has been transferred, meaning that the filter must store the data stream without forwarding it to the eventual recipient until it's all been received. The storage performance required would be incredible, and the compuational requirements of store-and-forward gatewaying all Australian web browsing etc is incomprehensible.

Reasons why Microsoft Access absolutely stinks as a DB UI RAD tool

2010-04-16T14:56:00.001+08:00

When used as a UI builder & RAD tool for accessing a real database (in this case PostgreSQL), Microsoft Access absolutely stinks. It has some severe problems that require cumbersome workarounds. In fact, if you're vaguely competent with a programming language it's almost certainly better to just write the application in your preferred language and be done with it. Java + Swing + Hibernate, for example, would do the job very well, as would C++ & Qt .
I was forced to use Access in this most recent project, and thought I'd post a warning for others who might be considering it but have alternatives available.
Note that none of this applies if you use MS SQL Server as the backend since Access doesn't use ODBC linked tables for MS SQL Server, but rather its own custom interface that's much smarter and more capable.
Problems using Access with ODBC linked tables include:

It doesn't understand server-side generated synthetic primary keys. It can't just insert a record with a DEFAULT key and then retrieve the generated key through the ODBC driver. To use a PostgreSQL SERIAL column (a sequence) you must write an event procedure in VB that fires on the Form_BeforeInsert event. This procedure must issue a passthrough query to invoke nextval('seqname'), then store the value in the primary key field of the form. Hooray for a tool that does everything for you transparently, simply, and automatically.
When you write your passthrough query, you will notice when examining the server query logs that it fires twice. That's because, if the ReturnsRecords property is set on a query, Access appears to issue the query twice. Presumably it's using the first instance to enumerate the returned fields for type-checking purposes, as it does not do this when the ReturnsRecords property is false. In any case, this means that you can't get the return value of a stored procedure with side effects without invoking the procedure twice. So, for nextval, I've had to first call nextval('seqname') and discard the return value, then call currval('seqname') to get the return value. That's two round trips to the database that were completely unnecessary, and a total of four for a single INSERT.
Access has no concept of data dependencies between fields. If one field is (say) a combo box populated by a query that depends on the value of another field, you must use an event procedure in VB to trigger a refresh of the dependent field when the field it depends on is changed. This would seem to be one of the more obvious and basic things a RAD database tool would do for you, and I was stunned that it does not.
Access loves to retrieve huge amounts of unnecessary data. If you open a form that's bound to a linked table, it'll issue a query like SELECT pkey FROM table;, which will result in a sequential scan on the whole table for databases that can't return the data from the index (like Pg¹). Even for databases that can, they must still read and scan the whole index. That's also potentially a lot of network I/O. You can work around this with EXTREMELY careful use of filter rules, but you have to be quite paranoid to make sure it never sees the form/table without a filter rule.
Access cannot natively perform non-trivial queries server-side - you have to write them yourself with Visual Basic and use unbound forms, which isn't too dissimilar to how you'd be working with (say) Hibernate, just uglier. With a bound form and a linked table, Access will use a WHERE clause on the primary key when fetching a record, but that's about it. It can perform simple inner joins server side too, apparently, but I've hidden everything I can behind server-side updatable views so I haven't had to struggle with this aspect yet.
Access cannot handle NULL boolean values.. Use a smallint with a constraint on it instead.
Access needs some custom operators and casts defined to work with PostgreSQL.. It assumes that certain types may be cast to other types, compared to them, etc in ways that aren't allowed by default. Thankfully PostgreSQL supports user-defined operators and casts, so you can just add them. Here's one page with a useful set of operators and casts..
Access likes to test for nullity with the = operator.. This is just wrong - the result of NULL = NULL is NULL, not TRUE, since the two unknown values may or may not be equal (the result is also unknown). PostgreSQL has a hack parameter, transform_null_equals, that may be set to enable the incorrect behaviour that MS Access expects. Access MAY have been fixed in version 2007; I haven't had cause to test this particular quirk yet.

You can work around a lot with careful use of filters, lots of Visual Basic code, plenty of views, updateable views, and stored procedures, and by writing your stored procedures to have EITHER side effects OR a return value (never both). It's not much fun, though, and it's still far from efficient.
On the upside, with row versioning enabled in the PostgreSQL ODBC driver, Access uses the MVCC attributes of the tuples (xmin, etc) to do optimistic locking. This allows it to avoid long-running transactions and long-held locks during user interaction without having to mess with the schema to add a field for oplocking use. It'll even play well with the oplocking used by things like Hibernate, and will work properly with apps that use ordinary transactional locking (SELECT ... FOR UPDATE and so on) without the need for any sort of support triggers. It's very nice.
To use Access with PostgreSQL, you REALLY need to set these options in the ODBC DSN for the connection:

Row Versioning causes the driver to use only the primary key in WHERE clauses, and to use xmin & affected record count during updates to avoid conflicts.
True as -1
(unchecked) Bool as Char

¹ This isn't as stupid as it sounds. PostgreSQL uses an MVCC approach that allows for very efficient, highly concurrent inserts and selects. The price is that it must store tuple visibility information to know which transactions should see which tuples, since the table contains the data at many different points in time all at once. Visibility information isn't in the indexes (because it'd generate lots more write activity on the indexes, slowing down I/O) so once an index indicates a match on a tuple, it must be read from the heap to check visibility even if the index contains the actual data point we want. It's a performance trade-off that is not particularly great when you're doing a `count(*)' or `select pkey from table' but is wonderful on highly concurrent systems.

Nice work, NetworkManager

2010-04-16T14:29:00.002+08:00

I'm yet to encounter a cellular modem that NetworkManager 0.7 (in Ubuntu 8.10 and 9.04-beta) doesn't automatically recognize without any user configuration, driver installation, or anything. Just plug it in (if not built in) and start using it.

Very nice work.

Dongles are EVIL

2010-04-16T14:27:00.003+08:00

The device you see on the right is actually the devil. Or, at least, it's close enough if you are a system administrator.

It is a single piece of hardware that controls your access to business-critical programs. Lost the dongle? Whoops, no classified ads in the newspaper this week. Dongle broke? Ditto. Dongle fried by a computer malfunction or power fault? Ditto. Computer stolen? Ditto.

What's even more fun is that as computers move on and older interfaces become obsolete, it becomes hard to even find a computer you can plug the dongle in to. Most machines don't have parallel ports anymore, so parallel dongles like this one are a big problem. At least that can be worked around using USB adapters.

Of course, then you run into exciting issues like XP being unable to allow 16-bit code access to the parallel port. The program would work fine on XP, but for the stupid bloody dongle. So you're forced to maintain legacy hardware or waste time on complex emulation/virtualisation options just to get the program working, when it'd be just fine but for this dongle.

So, if you are ever offered software for any reason that requires a dongle, just say no.

Bought to you by the exciting battle to get an old and alas mission-critical win16 app to work under WinXP or even WINE.

Getting GNOME Evolution to offer a client certificate for IMAP SSL/TLS

2010-04-16T14:25:00.008+08:00

GNOME Evolution isn't noted for its client certificate support. Entries in the bug tracker about it have rotted for years, and it has absolutely no acknowledged support whatsoever. Most other mail clients have had client cert support for years if not decades.

Unfortunately, Evolution is quite attractive in other ways - calendar integration, LDAP address books, etc. Unlike Thunderbird (especially when large images are involved) it also has acceptable performance over remote X11 connections.

So - I'd rather like to be able to use Evolution, but it's client support ... isn't.

It turns out, though, that Evolution uses the Network Security Services library from Netscape/Mozilla . It's used, among other things, for IMAP SSL/TLS support. This library does support client certificates; after all, Thunderbird and Firefox support client certificates and they do their crypto through NSS.

Is it not then possible to introduce a client certificate at the libnss level, so Evolution doesn't even know it's doing client certificate negotiation during its hand-off to NSS for SSL/TLS setup?

Why, yes, it is, and it takes one line of code in camel-tcp-stream-ssl.c to do it.

camel-tcp-stream-ssl.c:
- /*SSL_GetClientAuthDataHook (sslSocket, ssl_get_client_auth, (void *) certNickname);*/
+ SSL_GetClientAuthDataHook (ssl_fd, (SSLGetClientAuthData)&NSS_GetClientAuthData, NULL );

Because Evolution its self still has no idea about client certificates, if the server demands one and you don't have one installed you'll still get a useless error message instead of an appropraite prompt to install a client certificate. Just like Thunderbird and most other client-cert supporting apps. However, if you install a client cert by importing it into the Certificates section of the preferences, evolution (or more accurately libnss) will present it and use it when the server asks for it.

Update late 2009:

Committed in stable (gnome 2.28.1+) http://git.gnome.org/cgit/evolution-data-server/commit/?h=gnome-2-28&id=87238717ceb0a158a00c76fc07c6e27c769c2cf0
Committed in master (gnome 2.29.1+) http://git.gnome.org/cgit/evolution-data-server/commit/?id=429a106d101bf205ba0c8ee8f94a818327c2d736

Update mid 2010:

This code has now hit shipping Evolution versions in up-to-date distros like Ubuntu 10.04 and Fedora 13. I've tested it in Ubuntu 10.04 and verified that client cert support works now. Hooray!

Getting central certificate management working on modern Linux

2010-04-16T14:23:00.000+08:00

Modern Linux (GNOME, anyway) systems actually have a central certificate store. It's a bit lacking in management UI so far, but it works, and you can use it instead of loading your PKCS#12 certificates into every app you use manually.

First, import your certificate into the GNOME keyring with:

gnome-keyring import /path/to/certificate.p12

Install the libnss3-tools package (containing modutil).

Now exit every application you can, particularly your browser and mail client. Kill evolution-data-server too.

Find all instances of the nss security module database on your homedir, and for each one (a) test to make sure it's not open and (b) install the gnome-keyring PKCS#11 provider in it. The following shell script snippet will do this for you. Just copy and paste it onto your command line:

for f in $(find . -maxdepth 5  -name secmod.db -type f  2>/dev/null ); do
  echo "Testing: `basename $f`"
  if fuser `dirname $f`/cert8.db >&/dev/null; then
    echo -n "In use by: "; fuser `dirname $f`/cert8.db; echo " - Skipping"
  else
    modutil -force -dbdir `dirname $f` -add GnomeKeyring \
            -libfile /usr/lib/gnome-keyring/gnome-keyring-pkcs11.so
  fi
done

Now all your NSS-based apps should know about gnome-keyring and use the gnome-keyring certificate store.

If you use Evolution and want client certificate support, patch evolution-data-server as per GNOME bug 270893 to enable that too. It'll use gnome-keyring automatically.

i1Pro and the Australia Tax

2010-04-16T14:20:00.000+08:00

The X-Rite i1Pro is an important instrument for anyone doing serious computer colour work, particularly for print and pre-press. It's also incredibly pricey, especially if you happen to live in Australia.

There's this oddity known locally as the "Australia Tax". The Australian Tax Office may not know about it, but it appears that local distributors for international businesses are convinced that it exists, and that it's high. That's the only explanation I can find for some of the jaw-dropping price differences between US and Australian versions of the same products - most of which are made in China anyway.

I got the X-Rite i1Pro instrument cheap (ish) - at AU$1500 ex GST and shipping compared to the AU$1800 quoted price. X-Rite force you to buy through exclusive local dealerships that add a huge markup, so while the US price is US$995 for the same instrument (AU$1200 @ current rates) you can't just order from the US. They won't ship it to you. You can use a US remailing service but X-Rite won't register it and won't support it outside the US - and neither will the AU distributor. You can't get it recalibrated etc without a painful amount of effort.

The AU dealership tries to claim it "adds value" ... but they don't do local advanced tech support, don't have any techs or offices outside metro Sydney, ship the instruments off to the US (3-4 week round trip) for calibration, and don't even keep spares in stock. So what value, exactly, do they add, other than to the price tag?

X-Rite and their distributors are raking it in with this arrangement. Unfortunately, X-Rite have been buying out all their competitors (like GretagMacbeth) so they're the only game in town. Like Quark, they'll suffer for their customer-hostile attitude and parallel import restrictions eventually, but right now they're in the "raking in the dough" phase.

(Of course, literally three days after I bought the i1Pro, Graham Gill, who develops Argyll CMS, announced support for the much cheaper ColorMunki spectrophotometer ... but hey, the i1Pro is a much better instrument so no harm done.)

Client certificate WTF

2010-04-16T14:08:00.002+08:00

Why does almost nobody bother to support X.509 client certificates properly? They're a weak, poorly implemented afterthought in many systems if they're supported at all.
(Note: Android info here may be dated, as I last tested with 2.0, and applies only to the stock Android distribution not apps or patched versions supplied by vendors.)

Microsoft Windows: Perfect support, though it requires PKCS#12 (.p12) format certificates. Most 3rd party apps (subversion, mozilla apps, etc) use own cert stores rather than the OS's for no good reason.
Mac OS X: Limited support in OS keychain. No support in OS services like Apple Mail (IMAP+TLS, IMAPs, SMTP+TLS or SMTPs), WebDAV over HTTPs, etc so I have no idea why they even bothered adding support to keychain. Some apps have their own support, eg Mozilla apps via NSS, but OS has none and most Apple apps have none. No 3rd party apps seem to look in the system keystore for certificates.
Linux systems: All major SSL/TLS libraries have support, but there's no system-wide or desktop-wide keystore or key management. Netscape security suite apps have good support but certificates must be installed individually in each app - even though the underlying libnss library supports a shared user-account-level security store. GnuTLS- and OpenSSL-using apps must implement their own certificate management but can support client certificates if they provide appropriate callbacks. As a result, support is very app dependent and often buggy or very clumsy. Real-world support is inconsistent - eg Subversion supports client certs, but many svn front ends don't handle the cert prompts; Thunderbird & Firefox support client certs via NSS; Evolution supports via NSS but has broken nss init code that breaks client certificates and is often built with GnuTLS instead of NSS anyway; etc. Overall it's painful but usually usable.
Symbian (Series 60) phones: Support is perfect in OS and apps. Very smooth.
Sony Ericsson phones: Seem to have no concept of client certificates, and treat request for client cert by remote mail server as an SSL/TLS error.
Windows Mobile phones: Basically perfect from all reports. Pity about the rest of the OS.
Apple iPhone: Decent client cert store support. Unclear how much access 3rd party apps have. It's used for safari. Reported broken in Mail (comment by Martin).
Android phones: are a near-total information void. Apparently it's just assumed you'll use Google's services, not (say) your own secure mail server with your work. Because, you know, who needs confidentiality anyway? If you download the SDK and phone emulator, you'll quickly find out that not only does the OS lack any way to import a client certificate or use one in negotiation, but it lacks any way to even import new CA certificates. That's stunningly, jaw-droppingly pathetic. Of course, this is a phone with a read-only IMAP client so it's not clear what, exactly, it's meant to do...

Sigh.

Update: It looks like there's half-baked and mostly user-inaccessible support for importing CA certificates in some flavours of Android. This app exploits the facility. Client certificates don't appear to be so blessed.

Google Android is not a smartphone OS

2010-04-16T13:59:00.000+08:00

... it's a simple phone OS plus a web browser and some Google services. At least if the devkit phone simulator I used to see if this was something I might want to actually use is anything to go by.

It appears to lack some pretty fundamental things you'd expect from a smartphone.

Ability to browse and view local files on phone memory or SD card, eg open HTML files, PDFs, etc
An IMAP client that can delete messages, mark them as read on the server, etc
Any ability at all to support corporate private CAs (Certificate Authorities), since it can't import new X.509 CA certificates
Any X.509 client certificate support for secure mail and intranet access
Decent sync and backup facilities to a laptop/PC. Oh, wait, you only use Google services, right?

This is Google's fancy new phone platform? Call me again in a few years, once you've grown up a bit - right now, even the iPhone OS is a more solid choice.

The best eBook reader for Linux is currently....

2010-04-16T13:56:00.000+08:00

Microsoft Reader run under WINE.

Sigh. Not only is it the best, it's practically the only one unless you're content with fixed-format PDF. Few eBooks are available in, or reasonably convertible to, HTML, and even if they were there aren't any HTML renderers that can do half-decent H&J. None at all can hyphenate even poorly, and justification support tends to be limited to clumsy expansion-only justification that is ugly and not very nice to read.

So, to get a decent result one would basically have to hand-convert a plain text or HTML format book (possibly after pdf-to-text conversion) to TeX and typeset it for a particular display. That's not exactly a nice, convenient way to sit down for a good read. Even then, unless you use pdftex and read with Adobe Reader it won't even remember your place!

By contrast the Microsoft Reader .lit format is fairly widespread, supports automatic and somewhat decent H&J (though nothing on TeX / InDesign ), remembers your place in each book in the library, tracks and manages the library without forcing a particular on-disk structure on it, supports easy drag-and-drop of a book onto the program even from Nautilus, etc. It's friggin' emulated* Windows software that hasn't been updated or improved since it was practically abandoned by MS in 2003 and it's still better than anything available natively for Linux.

The situation is just as dire for Linux-based ( and Symbian-based ) phones and tablets. Given the spread of Qt to more and more devices, as well as all major platforms, I'm increasingly tempted to start work on a Qt-based reader with decent H&J, library management, place tracking / bookmarking / margin notes, etc. But how can there not be something out there already? Am I just blind, or is there really a gaping hole this big in free software capabilities?

Any suggestions? Anyone interested in working on one?

* I know, I know; I just don't care that W.I.N.E. Kudos to the WINE folks for their amazing work.

Australian Bandwidth Pricing

2010-04-16T13:53:00.001+08:00

I recently switched hosting of large files (12Mb or so PDFs) on my employer's website from the existing Australian host to one in the USA. Why? Because it's cheaper to send data from the USA to Australian users than it is to send it from within Australia..

About 100x cheaper, in this case, when comparing Anchor Networks per-Gb pricing to SimpleCDN's.

SimpleCDN doesn't have any Australian node(s). Data gets requested by their US nodes on first request, cached, and sent back to Australia. Yet they're incredibly, vastly cheaper than anything local I can find.

The root of the problem appears to be that Australian hosting providers charge all IP traffic as if it were to go via an international link. There's no provision made for peering or intra-national traffic at the majority of hosts. This may be an issue with the hosting provider its self, or it may be with their upstream bandwidth suppliers, but I don't care. Internet routing is designed to solve this sort of problem - thankyou BGP - and peering points exist for a reason.

It's actually way cheaper to store your data in the UK, Ireland, Germany, France, or pretty much anywhere except Australia even if your users are 100% Australian. Isn't that kind of sad?

Beware of soapy frogs

I've had it with HTC - thanks for the rescue, CyanogenMod + AAHK

Atmel ATtiny with built-in 433MHz (US: 310MHz) transmitter!

DIY DC soil moisture sensor - early test successful

First step: Simplest of the simple DC resistive sensor

Next step: AC resistive sensor

Interested in soil moisture sensors and irrigation control? Start with the UF/IFAS virual extension series

Using a RHT03 (aliases: RHT-22, DHT22, AM2302) temperature/humidity sensor from Arduino

Extending Arduino example CIRC-05 to use hardware SPI control

SparkFun Inventors Kit CIRC-03 - Motor not working (spinning)? It's an error in the instructions

Windows command line survival: findfiles, find+xargs's dim cousin

Bulk conversion of OpenType (OTF) to TrueType (TTF) or Type 1(PFA/PFB) fonts using FontForge

PostgreSQL: Great even when you can see the warts

No further work on Kobo

Kobo's product isn't the hardware

Taking patches and dealing with random "developers" can be lots of work

The plugin API was a non-starter

So it's really not that simple

Java EE 6 - Traps, pitfalls and warts list

CDI vs legacy

JSF2 and JAX-RS have incompatible access methods for HttpServletRequest etc

JNDI naming is confusing and hard to debug

Related to the above: persistence.xml ignores resource-ref mappings

Configuration and deployment descriptors are within the application archive

Glassfish 3.0 was released prematurely, and has buggy CDI until 3.1.1

Someone else in Java EE 6 pain

Java EE application servers - learning from the past

Background: classloader leaks

Lessons from the past

Applying the lessons to EE app servers

Database preferences and product selection methodolgy

Product selection, values, and selection methods

Support

Pg as a poor Oracle clone

In conclusion

Solo sysadmin/coder

Using Seam 3 with Glassfish 3.1

Update Weld (the CDI implementation) in Glassfish

Work around Glassfish's over-strict class scanner

Weird BeanManager exceptions

Scribd is amazing

Useful things I've discovered about Glassfish (especially embedded glassfish) lately

Glassfish properties

Creating users in a file realm

maven-embedded-glassfish-plugin

Documentation for the embedded glassfish 3.1 api

asadmin from embedded

REST admin

JavaServer Faces (JSF2) and JAX-RS don't play well together

Documentation

JSF2 doesn't mesh well with JAX-RS

Results

File upload webapp

Kobo and long-unfixed bugs

(You can't) Go To Page (fixed in 1.9)

Reading position loss sucks

Which of the set I just added scrambled its brains?

I'm not currently reading that (fixed in 1.9)

Dammit, which is the next book?

Quote me once

Don't just whine about it, do something!

Drupal 7, PostgreSQL, unserialize, and bytea_output

Handling file uploads with Java EE 6 / JAX-RS / Glassfish / Uploadify

Jersey 1.5 and jersey-multipart's @FormDataParam

Determining the correct path to the upload handler

Handling form data on a page that accepts Uploadify file uploads

How to get the context path (context root) in jsf2

Setting up usb gadget serial (g_serial) on Kobo Wifi

Compiling Qt plugins for Kobo Wifi

Taking a disk image of the Kobo Wifi without opening the device

Imaging the Kobo's firmware over wifi

Imaging the Kobo's firmware to an SD card

Examining the firmware image

The Kobo's kernel

Enabling telnet and ftp access to the Kobo Wifi

Changing the Kobo's boot image

Preparing a development environment for the Kobo Wifi

Overview

Installing Sourcery G++

Compiling libraries

`asadmin` from embedded

Jersey 1.5 and jersey-multipart's `@FormDataParam`