Saturday, February 26, 2011

Kobo and long-unfixed bugs

This article is part of an extended series on Kobo development and investigation


Apparently I have to make this more obvious:

  • Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
  • Any device hacking is risky. If you can't afford to break your device, leave now.

(This post is part of a series on the internals of the Kobo and Kobo Wifi eReaders. The rest of the posts are technically focused; this one was just prompted by increasing frustration with the device.)

I really like the Kobo. It has great hardware, a great price, and some pretty nice software running on it. Unfortunately, it also has some annoying bugs and limitations - many of which wouldn't be too bad,if only there was any sign they'd ever get fixed/resolved. some but not all of which have now been fixed in the 1.9 firmware.

I feel a little bad paying the Kobo out about these issues, because the Kobo folks were really great to deal with and really responsive early on. I've been absolutely blown away by how far beyond narrow-interpretation GPL requirements they've gone with their source releases (including full build scripts and config files) and how quick they were to update their releases when asked about them. Please keep in mind that overall the Kobo is an excellent eReader and the Kobo guys deserve our support for being one of the depressingly few companies who fully accept, understand and act on the obligations that fall on them in exchange for the free use of GPL-licensed software.

Friday, February 25, 2011

Drupal 7, PostgreSQL, unserialize, and bytea_output

I just found out that PHP 5.3's PostgreSQL PDO driver (as used by Drupal) is broken by the Postgresql 8.0 transition from octal to hex encoding for bytea. It passes the raw hex through to the app, including the leading x , causing PHP's unserialize() function to choke.

I'm using Drupal with Apache and PostgreSQL on Windows (sigh) because I'm writing up a step-by-step hand-holding guide for someone who needs to do some testing against our Drupal database. I wouldn't be using that configuration voluntarily ;-)

Drupal doesn't check that bytea_output is set to 'escape' as a workaround or do a sanity test to detect this fault, so the results are ... interesting:

Notice: unserialize(): Error at offset 0 of 27 bytes in variable_initialize() (line 749 of C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\drupal-7.0\includes\bootstrap.inc).

If anybody else lands up whacking their head against this for a while: just

ALTER DATABASE drupal SET bytea_output = 'escape'

to have your sanity restored.

The amount of breakage being seen in drivers really makes me wonder if the the octal to hex transition should've been done with an opt-in from drivers during negotiation or an explicit SET, rather than just making it globally default. But, hey, hindsight.

Friday, February 4, 2011

Handling file uploads with Java EE 6 / JAX-RS / Glassfish / Uploadify

UPDATE: This article is very obsolete. RichFaces 4, among other framworks, now handles multi-file upload out of the box. Use that instead.

Joeri Sykora wrote about using Jersey extensions to JAX-RS to handle file uploads in Java EE 6. His example is extremely handy, but needs some updates to handle Jersey 1.5.

Update: I've now put together a complete and self contained Java EE upload application using Uploadify, which you can grab from github.com/ringerc/postupload. See http://blog.ringerc.id.au/2011/03/file-upload-webapp.html

How to get the context path (context root) in jsf2

Use:

#{facesContext.externalContext.request.contextPath}

For some insane reason, none of the methods that should work would work when invoked via a @RequestScoped object with @Inject @FacesContext - they all merrily returned the empty string. More stupid CDI/JSF brokenness.

Monday, January 31, 2011

Setting up usb gadget serial (g_serial) on Kobo Wifi

This article is part of an extended series on Kobo development and investigation


Apparently I have to make this more obvious:

  • Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
  • Any device hacking is risky. If you can't afford to break your device, leave now.

The Kobo Wifi (apparently) has an onboard serial port, but it doesn't have any header pins let alone a usable socket. Using it will require bulldog clips at best, more likely soldering some pins in. Either way you have to open the case and keep it open while using the port which is inconvenient if you like to actually use your Kobo. In any case, I can't find any documentation for the pinout.

Telnet over wifi works and can be enabled without opening up the device - but tends to go down quite a bit, as it's not really intended for this use. It's hopelessly unsuitable for running a GDB remote debugger over.

It's fairly simple to modify the Kobo Wifi (and presumably the original Kobo, though I haven't tried it) to add support for serial-over-usb and/or ethernet-over-usb, both of which are much easier to work with than physical serial and much better than telnet/ftp over wifi. Enabling USB serial gadget support seems to interfere with the USB gadget mass storage system used to export the file system to a host computer, though, so don't make the change permanent unless you like to keep your library on an SD card. It looks like the 2.6.33-rc1 and newer kernels may contain support for multiple gadgets running at once, but the Kobo is on 2.6.28 and it's unlikely to be worth the effort of an update.

To get usb serial mode working, you must build a new kernel for the Kobo and copy the modules from that kernel over to the Kobo. You might want to make menuconfig and enable CONFIG_USB_CDC_COMPOSITE in drivers -> usb support -> usb gadget -> CDC Composite Device if you'd like to have simultaneous support for Ethernet and serial gadget mode, which can be really handy for debugging. See documentation/README.kernel in the KoboLabs git repository for how to build a new kernel and modules. I've asked them to pull it, so it should be there soon. You should not actually need to install the new kernel, as the modules you build should be compatible with the old kernel already on the device.

To install the modules, install them to some temporary MOD_INSTALL_PATH like /tmp/arm then cd /tmp/arm and tar cvzf KoboRoot.tgz lib. Copy the KoboRoot.tgz file to .kobo/KoboRoot.tgz on the device's onboard user-accessible flash storage and reboot to install the update. After a reboot, you can telnet in and modprobe g_serial (or g_cdc if you compiled the mixed serial/ethernet module) to enable USB serial support. To make the serial port useful you need to run /sbin/getty -L ttyGS0 115200 vt100 to listen for logins. This is easily wrapped in a shell script you can invoke via telnet, or via an autorun hook you add in /etc/init.d/rcS that runs a script off an add-in SD card if one is found. The same script can be used to bring up the usb0 ethernet device if you're using g_cdc.

If you want to make usb serial gadget support start at every boot (thus disabling the ability to manage the internal card library over USB - this will be annoying!), add this line to /etc/inittab on the device:

ttyGS0::askfirst:/sbin/getty -L ttyGS0 115200 vt100

... and add this line to /etc/init.d/rcS after "/bin/mount -t sysfs" :

/sbin/modprobe g_serial

If you'd prefer to have serial and ethernet, and made the kernel config change to enable it as described above, replace g_serial with g_cdc in the line above.

Once you've loaded the g_serial module one way or another and started a getty, you can connect the Kobo to your PC. When prompted for what to do, say "keep reading". You'll discover that the Kobo appears as a USB serial port that you can connect a terminal to. On a Linux host it should be /dev/ttyACMx (/dev/ttyACM0 if you have no other USB serial ports); check dmesg to be sure. You can use a terminal program like minicom, gtkterm or picoterm to talk to the port. The login is "root" and there is no password unless you set one later. You can transfer files using zmodem's rx command if you need to, use the usb ethernet module, or ftp them over wifi.

Unlike telnet over wifi, this serial console stays up no matter what antics nickel performs, and doesn't require the device to be open. It's similarly easy to use g_ether to expose an Ethernet interface or g_cdc to do both ethernet and serial at once. It's just a real pain that using them prevents the g_file_storage module used by the Kobo to manage the onboard memory from working!

Friday, January 28, 2011

Compiling Qt plugins for Kobo Wifi

This article is part of an extended series on Kobo development and investigation


Apparently I have to make this more obvious:

  • Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
  • Any device hacking is risky. If you can't afford to break your device, leave now.

The Kobo Wifi's main user interface application, nickel, is a Qt Embedded app. It drives the ePaper display via a Linux framebuffer interface and some ioctl calls to trigger display refreshes. This is abstracted away from the app behind a QWS (Qt Window System) driver called broadsheet_ioctl that's statically linked into nickel.

Because the QWS driver is only in nickel, we can't really write stand-alone Qt applications to run on the Kobo without reimplementing it. However, the nickel app has a simple plugin interface you can use to load your own code as a shared library.

To do this, you need a full Kobo development environment, including a cross-compiled Qt Embedded build that matches the one used on the Kobo. Once you have that, you can build apps according to the template provided by examples/poker, then drop the shared library produced into /usr/local/Kobo on the device to get it to load.

You now have some totally useless code running on your Kobo. Congratulations. There's no way to invoke it. For now, I'm testing by replacing the poker plugin with my own code, but it's still painful to access it so it's not desirable for real-world use. It may prove necessary to introspect the Qt widget tree or even runtime patch classes to get useful functionality loaded by plugins without having access to the nickel sources.

If your plugin doesn't load, it may be helpful to telnet into your Kobo and create a script like this, called (eg) /usr/local/relaunch-nickel.sh

#!/bin/sh
# Kill off the old nickel
pkill nickel
# Clear plugin cache
rm -f /mnt/onboard/.kobo/Trolltech.conf
# Restart nickel
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib:
export NICKEL_HOME=/mnt/onboard/.kobo
export LD_LIBRARY_PATH=/usr/local/Kobo
export QWS_KEYBOARD=netronix
export INTERFACE=eth0
export LANG=en_US.UTF-8
QT_DEBUG_PLUGINS=1 /usr/local/Kobo/nickel -qws -display broadsheet_ioctl >/mnt/onboard/.kobo/nickel.log 2>&1 &

When executed, this script will restart nickel, causing it to write a debug log to the .kobo directory on the user flash and print detailed information about plugin loading. This might help you figure out what's going on. nickel prints a fair few warnings during normal startup, so don't be thrown by that.

When developing for Qt on Kobo, the Qt for Embedded Linux Environment Variables document may be helpful, though it doesn't mention QT_DEBUG_PLUGINS.

It can also be helpful to determine how Qt was built on your target device. On the Kobo Wifi:

$ strings libQtCore.so | grep 'Build key'
Build key:           arm linux g++-4 full-config

Thursday, January 27, 2011

Taking a disk image of the Kobo Wifi without opening the device

This article is part of an extended series on Kobo development and investigation


Any attempt to modify your device is risky. Your device might be permanently destroyed. If you cannot risk the permanent loss of your device, so not attempt to modify it. Something can easily go wrong if you do the wrong thing - or follow out of date instructions. I don't continually maintain and test these instructions, so they might well qualify.

If you can't afford to lose it forever, physically open your Kobo up before doing anything, remove the SD card, and take a disk image of it using a PC with a card reader. I use `dd` with a Linux box to make sure I get a complete image of the entire card.

Note that you could break the case or retaining clips when opening your Kobo. I didn't and I opened mine quite a few times, but who knows if they've changed assembly techniques, plastics, etc since then.

Messing with embedded devices that aren't designed to be hacker-friendly is risky. Deal with it, or don't mess with the device.


Once you've enabled telnet on your Kobo Wifi, you can use it to transfer a disk image of the Kobo's entire firmware image on the internal MicroSD card. This will give you a backup copy of your Kobo's firmware on your computer.

You can't restore the firmware image without physically opening your Kobo if you mess things up badly enough for it not to boot, but hopefully you won't do that. The Kobo seems to be amazingly hard to break unless you corrupt its file systems, overwrite its kernel, etc. Anyway, this way you can avoid opening the Kobo until/unless you ever need a total firmware restore.

Imaging the Kobo's firmware over wifi

To image the disk of a running system, we need all its file systems in read only mode. This turns out to be trivial to achieve with the Kobo, though it requires terminating the main eReader process ("nickel") so you'll want to reboot after you've done it. During the process, the screen will display whatever was on it before you ran the commands to image the system and none of the keys will respond. Just:

pkill nickel
mount -o ro,remount /mnt/onboard
mount -o ro,remount /
nc -l 9984 < /dev/mmcblk0

Now, take note of the IP address printed by the commands above and use it in this command on your main computer:

nc IP_OF_KOBO 9984 > kobo_internal_microsd.img

The disk imaging process will take a while over wifi, and won't show any progress. Mine took about twenty minutes. If you want progress indication, open a second terminal and run:

watch "du -ms kobo_internal_microsd.img"

... to get a count in megabytes of data transferred so far.

Finally, reboot the kobo to bring it back to normal. You can do this by pressing and holding the power button for 6+ seconds, or by typing control-C then "reboot" and enter into the telnet command line on the kobo.

The image saved on your computer should be 1977614336 bytes for a Kobo Wifi with a 2GB internal card (~1.1GB user accessible memory).

If you have any issues, try a different port number with nc.

^C
reboot

Once you have the Kobo's disk image, you can mount the partitions within it to examine them, and you can extract some individual components of the unpartitioned space in the card.

Imaging the Kobo's firmware to an SD card

In principle you can also image the Kobo's firmware to an SD card instead of using wifi. This will be much easier if you don't run Linux. You need a 2GB or larger card.

To image the Kobo's firmware to an SD card you can telnet in to the kobo and make the file systems read-only as above. Make sure you unmount the external sd card /mnt/sd if it is mounted. Then, instead of getting the IP address and running netcat in listen mode, you can just `dd' the internal mmc card to the external sd card, `sync' to force it to flush, and `reboot'.

Alternately, you may prefer to leave the external SD card mounted, remount everything else read only, then run:

dd if=/dev/mmcblk0 | gzip > /mnt/sd/kobo.img.gz

... to make a compressed image (slow!) on the file system on the SD card. This will be easier to work with under Windows, which doesn't work well with raw devices.

Examining the firmware image

gunzip the image if it's compressed. Either back up a copy of the firmware image that you will not touch, or make the firmware image read-only with chattr +i.

You may now mount the three file systems on the image using this little script - or by hand by using "fdisk -l" to dump the partition table, then calculating the offsets to pass to losetup. If you want to use my script, I make no promises that it won't eat your system and your cat, so be careful. Save the following to "kobomount.sh" and mark it executable:

#!/bin/bash
set -e -u
if test $# -ne 1 ; then
  echo "Usage: $0 firmware.img"
  exit 1
fi
IMG=$1
i=0
sudo -v
for offset in $(sfdisk -d kobo_2gb_microsd_image_v174.img | grep start | cut -d : -f 2 | awk '{print $2}' | sed 's/,//g' |grep -v ^0$); do
  sudo losetup --offset $(( $offset * 512 )) /dev/loop$i "$IMG"
  sudo mkdir -p /mnt/kobo/$i
  sudo mount -o ro /dev/loop$i /mnt/kobo/$i
  ((i++))||true
done

You may then aim it at your firmware image and it'll mount the contained partitions on /mnt/kobo/0 (the recovery partition), /mnt/kobo/1 (the main OS) and /mnt/kobo/2 (the user flash partition). All will be read-only.

You may also want to extract some blobs from the unpartitioned space at the start of the image. That's where the kernel, boot splash image, etc live. I don't have official documentation on the layout of this space, but reading the upgrade scripts in /etc/init.d suggests that these values should be about right. They are UNVERIFIED except for the boot splash image, which I've been able to edit and replace.

IMG=kobo_2gb_microsd_image_v174.img
dd if=$IMG of=serialnumber.bin bs=512 count=1 skip=1
dd if=$IMG of=hwconfig.bin bs=512 count=1 skip=1024 count=2
# The image that ships with the kobo is < 470 512 byte blocks long. It's not clear
# if something else takes the space between the end of the image and the start of
# the epson display binaries, if it's dead space, or if there's room for bigger
# images.
dd if=$IMG of=bootlogo.bmp bs=512 skip=1026 count=470 
dd if=$IMG of=epson_display_setup.bin bs=512 count=1 skip=1920 count=8
dd if=$IMG of=epson_waveform.bin bs=512 count=1 skip=1928 count=$((2048 - 1928))
dd if=$IMG of=kernel.bin bs=512 count=1 skip=2048 count=$((7564-2048))

The Kobo's kernel

We can now examine the existing kernel to learn more about it. This post tells us how to extract the kernel (but it's more easily done with scripts/extract-ikconfig from a kernel tree)

The hard way (which lets us extract the image, not just the config) in brief:

od -A d -t x1 kernel.bin | grep '1f 8b 08 00'
# Note the (decimal) offset, correct for offset of deflate header into line
dd if=kernel.bin bs=1 skip=0013076 | zcat > vmlinux

The easy way to dump the config:

scripts/extract-ikconfig $HOME/kobo/kobofirmwaredump/kernel.img

Boot loading is apparently performed by RedBoot (2.0 according to shipped sources), which should offer tools like `fis list' to report on the structure of the redboot-managed areas of flash storage. Building redboot is a nightmare, so I haven't investigated this yet.