Windows command line survival: findfiles, find+xargs's dim cousin

Sometimes, we have to use the Microsoft Windows cmd.exe command line. Maybe it's at a client site or on a user machine where installing PowerShell, Cygwin or Mingw isn't appropriate. Maybe you're on a domain member so locked down by group policy you can't do anything if you want to. Either way, you have to use cmd.exe, and you're swearing.

I find out about the odd useful command that makes the Windows shell more usable, and I'll be trying to post them here. The first is forfiles, a basic alternative to find -exec or find|xargs.

Bulk conversion of OpenType (OTF) to TrueType (TTF) or Type 1(PFA/PFB) fonts using FontForge

FontForge is a great tool for converting fonts. I needed to convert a lot of OpenType fonts to TrueType or Type 1 format to use them with Apache FOP, as FOP doesn't yet support OpenType fonts with CFF glyph formats. Doing the conversion in one FontForge script didn't work well, because (a) it leaked memory until it got OOM-killed and (b) it only used one CPU. So I wrote a helper shell script to control the job - posted here in case anyone else needs it later.

PostgreSQL: Great even when you can see the warts

I’m a very happy PostgreSQL user, and was interested to read a recent post by an MS SQL Server DBA advocating PostgreSQL.

This post is a response, enumerating areas where Pg may not be so hot after all. You might say it's my tendency to always see the negative, or perhaps it's just my desire to inform others of the downsides of something so the loud advocacy doesn't leave them feeling let-down when the reality hits. I'd like to think it's the latter. Telling someone how it is from the start is much better than showing them the rosy-goggles view until they're committed. So, from an enthusiastic PostgreSQL user who spends a lot of time helping out on the mailing lists, here's a list of some of the warts and quirks I'm aware of, so you don't have to find out about them the hard way:

This post is based on PostgreSQL 9.2. Each version improves a great deal, so don't be surprised if some of these issues are gone soon.

No further work on Kobo

This article is part of an extended series on Kobo development and investigation

I've stopped work on the Kobo software and dev env.

Java EE 6 - Traps, pitfalls and warts list

Java EE 6 has the potential to be a great platform. It just needs plenty of cleanup, implementation bug fixing, and real-world use to get there.

One of the bigger issues is the significant array of spec inconsistencies and oversights that leave parts of the system working (or not working) in ways very different to one might expect. These issues can greatly increase debugging time and take important time away from productive development and into chasing issues in appservers, frameworks, and the specifications themselves.

I thought I'd make a list of some of the ones I've hit so far. I'm really hoping we'll see a Java EE 6.1 - a fixes-only revision that, unlike EE 7, focuses on polish and usability over adding new features. These are things I think are important to see covered by it.

The shortlist, explained in detail below:

You can't @Inject into a @FacesConverter. Use Seam 3 Faces to work around this. UPDATE: This should be fixed in JSF 2.1.

You can't @Inject into a JPA 2.0 EntityListener or an entity. Do a direct BeanManager lookup from JNDI or use Seam 3 Solder's BeanManagerAware to kind of hack around this; it's ugly. Lack of JPA 2.0 lifecycle hooks for EntityListener classes means Seam 3 Persistence can't fix this one. UPDATE Jan 2012: It appears that into EntityListener will be supported in JPA 2.1, but not injection into entities.

Completely avoid all use of javax.annotation.ManagedBean. It was obsolete before release, is only inconsistently recognised, duplicates JSF2 functionality that's also obsoleted by CDI's @Named, and should simply be removed.

Weld (CDI) before version 1.1.1 is extremely buggy. If using Glassfish 3.0, 3.0.1, or 3.1 you need to update weld-osgi-bundle to 1.1.1. See these instructions for Seam 3. UPDATE April 2012: These weld/glassfish issues are largely obsoleted by newer releases.

Mapping of application-local resource-ref resource names in web.xml to a global resource name defined in glasfish-web.xml or jboss-web.xml doesn't work for persistence.xml jta datasource references and isn't supposed to work. Ignore all the examples and official documentation that suggest it should, they work with direct JDBC and with Spring but not with JPA 2.0.

If you encounter mysterious silent failures or null pointer exceptions, you probably forgot to include beans.xml. Your container will not warn you about this at deployment or runtime even if CDI annotations are present on classes.

Many of these workarounds require the use of Seam 3 (ie: "let's finish the work EE 6 started") modules. Because of Glassfish bugs, some additional work is required to use Seam 3 on Glassfish versions prior to the yet-to-be-released 3.2. See this previous post for some information.

In addition to the genuine issues with EE 6, there's also the learning curve to contend with. A while ago I wrote a broad conceptual overview of EE 6 intended for those coming to it cold, rather than migrating from EE 5 or from Seam. I felt it was necessary because of the struggle I had getting started with EE 6 and the bugs that bit me and confused me during the learning process. Perhaps it'll benefit others here or even provide insight into areas of the documentation and tutorial that would benefit from improvement.

Now, for the details:

Someone else in Java EE 6 pain

I'm not alone!

Harald Wellmann writes about his experience moving a Tomcat-based app to Glassfish, JBoss AS 6, and Resin. Warts and all.

I think Java EE 6 really needs more real-world attitude and less hype. More bug fixing and more spec tidying, less focus on the next whizz-bang feature. It's nice to see somebody else who's had similarly painful experiences with it. Perhaps I'm not stupid after all, I simply make the foolish mistake of expecting released software like Glassfish 3.0.1 to have most features fairly solid.

Harald also notes that CDI was a huge problem with Java EE 6 until quite recently, largely because Glassfish 3.0 was half-baked and under-tested; JPA 2.0 is prone to exciting implementation differences and bugs, and more.

My favourite quote:

Legacy

Backward compatibility is a good thing for veteran users of a framework. Users don't want to change all of their application code just for upgrading to a new framework version.

On the other hand, backward compatibility can be extremely confusing to new users: There's two or three solutions for the same kind of problem, the legacy one is not marked as legacy or deprecated clearly enough, tutorials and example code still use the old style, and you can only resort to your favourite search engine or to trial and error to make things work consistently.

(Emphasis mine)

That's a perfect description of some of the problems I encountered when trying to get a handle on the released standard Java EE 6 and Glassfish 3.0, which I foolishly assumed was a stable platform to build my app on as I moved into the Java EE world

Java EE 7 really needs to modularize the backward compatibility crap into separate archives so that apps can depend on either javaee-7 or javaee-7-legacy as appropriate. That way we'll finally be free of the two copies of the obsolete-before-release @ManagedBean annotation and all the other horrible legacy crap.

Java EE application servers - learning from the past

(Edit 15 July 2011: JBoss AS 7 is here, and brings a huge improvement to class loading and memory management. It's not full isolation, but it limits the exposed contact surface between app server and app greatly, and massively improves class loading. Brilliant!)

If you've used a Java EE application server like Glassfish or JBoss AS for long, you will have experienced classloader leaks, though you may not have realized it.

If you've ever seen the error java.lang.OutOfMemoryError: PermGen space at deploy-time, read the linked article above. If you have ever worked on the JVM, on app servers, or on EE applications, please read on.

Even if you haven't hit classloader leaks, you should be aware of the various ways Java EE applications can cause memory leaks in the server and what to do about them.

For those coming here for help fixing an immediate issue with their app: Read the links above, and this article on using jhat's JavaScript interface to find likely leaks. More fancy JavaScript OQL tricks are here.

Database preferences and product selection methodolgy

I recently stumbled across an interesting weblog post by a DBA who expresses a strong preference for Oracle over PostgreSQL. I thought I'd respond to it with a few thoughts here, not so much because of the opinion expressed as the reasons given for it.

Solo sysadmin/coder

Are you a solo coder who doubles as a sysadmin or some other role(s) in your organization?

I share your pain - and I want to share these articles with you, because if you struggle with it too they will change how you look at it.

• Coding Horror: In programming, One is the loneliest number
• Burn-out
• Depression, burn-out and writing code

The Coding Horror article was written by Jeff Atwood, a damn fine developer and co-founder of Stack Overflow / Stack Exchange, which to me gives it a bit of added oompf.

While you're at it, read this article on programmer's bad habits ... because you probably do at least one of them. I know I do. They're counterproductive and just make you feel worse, so being aware of them helps.

Of course, nothing is as good as saying "I'm dropping all my other roles, you can hire someone else for them. I'm just programming now, that way I can actually finish the projects I started two years ago." I only have to hold out another two months before the new guy at work is ready to take over :-)

Using Seam 3 with Glassfish 3.1

Seam 3 builds on top of the Java EE 6 standards, filling in holes and omissions in the functionality provided by the specs and providing important enhancements that many programmers would otherwise find themselves implementing themselves. Seam 3 Solder in particular addreses some of the frustrating limitations in the CDI and Java EE 6 specs, providing well-engineered and easily re-used solutions to common problems.

Being a JBoss project, Seam 3 is unsurprisingly better tested on the JBoss application server. However, its goals explicitly cover portability and it's supposed to work on Glassfish and - where possible - even on servlet containers like Tomcat and Jetty. Perhaps unsurprisingly, Glassfish doesn't get as much attention , so it's a bit harder to use Seam 3.0.0.Final on Glassfish than it is on JBoss. It appears that it was also harder for the Seam 3 folks to get quick fixes for the many Glassfish bugs they found into Glassfish than it was for them to get fixes into JBoss AS, so many of the fixes for issues found in Seam 3 won't hit a Glassfish stable release until 3.2 comes out, if then.

In the mean time, there are some quirks to work around. The Seam 3 project documents Glassfish 3.1 compatibility issues here and you should read that document before continuing.

Many of the issues are solved pretty easily once you know what to do and where to look. It's well worth using Seam 3 instead of rolling your own solutions to many of the problems it tackles, so read on.

Scribd is amazing

I've recently been looking for better ways to publish online editions of the newspaper I work for. We don't have a full content managed workflow internally, nor do we use automatic layout, so CMS-driven cross-publshing isn't currently an option. We're stuck with finding something to do with the PDFs that come out of the print production process for the forseeable future.

Scribd appears to be a much more impressive option than direct PDF publishing or tools like flexpaper & pdf2swf. The HTML5 viewer is amazing. Note how all the elements are selectable, columns have been correctly recognised in the pdf text, etc?

Here's an example.

Useful things I've discovered about Glassfish (especially embedded glassfish) lately

I've been working with / struggling with Glassfish a lot lately. The documentation leaves much to be desired, so there are a few bits and pieces you might not know that might come in handy.

These notes refer to Glassfish 3.1.

JavaServer Faces (JSF2) and JAX-RS don't play well together

I've been working with JavaServer Faces 2 (JSF2) since shortly after its release, and it continues to be an annoying mix of excellent and frustrating to work with. It's theoretically a great technology, especially with CDI (contexts and dependency injection) even despite the holes and quirks in the spec. There are just two wee problems.

First, unlike many of the other EE specs, the JSF2 spec really isn't written to be useful for developers intending to use JSF2 in their apps. It's very much a specification not a reference, and I found it hard to read even as a specification. Because I was getting started with JSF2 in the very early days of EE 6, I seem to remember that the now very helpful Java EE Tutorial wasn't EE6 ready yet or was incomplete, so I didn't have that as a basis for learning JSF2 and EE6 either. Perhaps my memory is just failing me. In any case, I often landed up stumbling through Google-land trying to find out details that I thought should be obvious and easy to find in the spec or other reference documentation. At the time, Google-land was full of outdated references to JSF 1, lots of migration guides from Spring to EE 6, and plenty of other things to side-track a confused newbie. All in all, it was really hard to get going with JSF2 alone.

Second, the JSF2 spec was clearly concocted with no thought of or co-operation with the JAX-RS spec, and it shows. They don't work well together, and it's really, really frustrating because they should be so useful in combination.

Edit: Part of the issue here appears to be that I was trying to use JSF2 just for the Facelets templating language, expecting to be able to interoperate with JAX-RS where possible. JSF2 is a framework and doesn't isn't really designed to enable you to easily step outside the abstractions imposed by the framework - not least because you shouldn't need to. I was probably going about this the wrong way in a square-peg, round-hole kind of way, and shouldn't have been trying to use JSF2 in the first place. That's not made particularly obvious by the docs, though.

File upload webapp

I recently became frustrated with the lack of easily self-hosted file upload app options. I needed to replace my employer's reliance on giant email attachments with something that wouldn't melt our mail server - and our clients could still understand how to use. This unfortunately ruled out the classic options like anonymous FTP.

In the end I wrote a new web application to accept files from clients over http and save them to a shared folder on the internal network. Many of the building blocks like uploadify were available, but there wasn't much around in terms of complete applications to do the job. Instead, people seemed to be using services like YouSendIt, Box.net, DropSend, etc ... which have downsides including:

• Inability to provide a canned recipient list, or high per-user per-month fees to support one
• Inability to customise the site's appearance and integrate it into your domain - without paying high fees for an "enterprise" version
• The need to download files once the client has sent them to the provider, often one-by-one via a clumsy web interface
• Windows-only client applications required to do batch downloads of files. Some services have Mac versions, but only one had a Linux client suitable for our advertising thin clients.
• Lack of a web services API - unless you pay high fees for the "enterprise" version.
• Lack of no-flash fallback upload methods
• Sometimes iffy browser compatibility

I wanted something really simple for clients - and staff - to use, that simply dropped files into a shared folder as they were uploaded and sent a notification email to staff when the upload was finished. Now that I've finished it, I'm releasing the result as open source software for the use of anybody who wants it. You can get it here:

https://github.com/ringerc/postupload

It's a Java EE 6 project that runs on Glassfish. Don't stress if you've never used any of that before, if you only know PHP, etc. The client-side is all familiar HTML and JavaScript, and the server side isn't especially complicated code. Detailed instructions for installing and running Glassfish are provided.

Kobo and long-unfixed bugs

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

(This post is part of a series on the internals of the Kobo and Kobo Wifi eReaders. The rest of the posts are technically focused; this one was just prompted by increasing frustration with the device.)

I really like the Kobo. It has great hardware, a great price, and some pretty nice software running on it. Unfortunately, it also has some annoying bugs and limitations - many of which wouldn't be too bad,if only there was any sign they'd ever get fixed/resolved. some but not all of which have now been fixed in the 1.9 firmware.

I feel a little bad paying the Kobo out about these issues, because the Kobo folks were really great to deal with and really responsive early on. I've been absolutely blown away by how far beyond narrow-interpretation GPL requirements they've gone with their source releases (including full build scripts and config files) and how quick they were to update their releases when asked about them. Please keep in mind that overall the Kobo is an excellent eReader and the Kobo guys deserve our support for being one of the depressingly few companies who fully accept, understand and act on the obligations that fall on them in exchange for the free use of GPL-licensed software.

Drupal 7, PostgreSQL, unserialize, and bytea_output

I just found out that PHP 5.3's PostgreSQL PDO driver (as used by Drupal) is broken by the Postgresql 8.0 transition from octal to hex encoding for bytea. It passes the raw hex through to the app, including the leading x , causing PHP's unserialize() function to choke.

I'm using Drupal with Apache and PostgreSQL on Windows (sigh) because I'm writing up a step-by-step hand-holding guide for someone who needs to do some testing against our Drupal database. I wouldn't be using that configuration voluntarily ;-)

Drupal doesn't check that bytea_output is set to 'escape' as a workaround or do a sanity test to detect this fault, so the results are ... interesting:

Notice: unserialize(): Error at offset 0 of 27 bytes in variable_initialize() (line 749 of C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\drupal-7.0\includes\bootstrap.inc).

If anybody else lands up whacking their head against this for a while: just

ALTER DATABASE drupal SET bytea_output = 'escape'

to have your sanity restored.

The amount of breakage being seen in drivers really makes me wonder if the the octal to hex transition should've been done with an opt-in from drivers during negotiation or an explicit SET, rather than just making it globally default. But, hey, hindsight.

Handling file uploads with Java EE 6 / JAX-RS / Glassfish / Uploadify

UPDATE: This article is very obsolete. RichFaces 4, among other framworks, now handles multi-file upload out of the box. Use that instead.

Joeri Sykora wrote about using Jersey extensions to JAX-RS to handle file uploads in Java EE 6. His example is extremely handy, but needs some updates to handle Jersey 1.5.

Update: I've now put together a complete and self contained Java EE upload application using Uploadify, which you can grab from github.com/ringerc/postupload. See http://blog.ringerc.id.au/2011/03/file-upload-webapp.html

How to get the context path (context root) in jsf2

Use:

#{facesContext.externalContext.request.contextPath}

For some insane reason, none of the methods that should work would work when invoked via a @RequestScoped object with @Inject @FacesContext - they all merrily returned the empty string. More stupid CDI/JSF brokenness.

Setting up usb gadget serial (g_serial) on Kobo Wifi

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

The Kobo Wifi (apparently) has an onboard serial port, but it doesn't have any header pins let alone a usable socket. Using it will require bulldog clips at best, more likely soldering some pins in. Either way you have to open the case and keep it open while using the port which is inconvenient if you like to actually use your Kobo. In any case, I can't find any documentation for the pinout.

Telnet over wifi works and can be enabled without opening up the device - but tends to go down quite a bit, as it's not really intended for this use. It's hopelessly unsuitable for running a GDB remote debugger over.

It's fairly simple to modify the Kobo Wifi (and presumably the original Kobo, though I haven't tried it) to add support for serial-over-usb and/or ethernet-over-usb, both of which are much easier to work with than physical serial and much better than telnet/ftp over wifi. Enabling USB serial gadget support seems to interfere with the USB gadget mass storage system used to export the file system to a host computer, though, so don't make the change permanent unless you like to keep your library on an SD card. It looks like the 2.6.33-rc1 and newer kernels may contain support for multiple gadgets running at once, but the Kobo is on 2.6.28 and it's unlikely to be worth the effort of an update.

To get usb serial mode working, you must build a new kernel for the Kobo and copy the modules from that kernel over to the Kobo. You might want to make menuconfig and enable CONFIG_USB_CDC_COMPOSITE in drivers -> usb support -> usb gadget -> CDC Composite Device if you'd like to have simultaneous support for Ethernet and serial gadget mode, which can be really handy for debugging. See documentation/README.kernel in the KoboLabs git repository for how to build a new kernel and modules. I've asked them to pull it, so it should be there soon. You should not actually need to install the new kernel, as the modules you build should be compatible with the old kernel already on the device.

To install the modules, install them to some temporary MOD_INSTALL_PATH like /tmp/arm then cd /tmp/arm and tar cvzf KoboRoot.tgz lib. Copy the KoboRoot.tgz file to .kobo/KoboRoot.tgz on the device's onboard user-accessible flash storage and reboot to install the update. After a reboot, you can telnet in and modprobe g_serial (or g_cdc if you compiled the mixed serial/ethernet module) to enable USB serial support. To make the serial port useful you need to run /sbin/getty -L ttyGS0 115200 vt100 to listen for logins. This is easily wrapped in a shell script you can invoke via telnet, or via an autorun hook you add in /etc/init.d/rcS that runs a script off an add-in SD card if one is found. The same script can be used to bring up the usb0 ethernet device if you're using g_cdc.

If you want to make usb serial gadget support start at every boot (thus disabling the ability to manage the internal card library over USB - this will be annoying!), add this line to /etc/inittab on the device:

ttyGS0::askfirst:/sbin/getty -L ttyGS0 115200 vt100

... and add this line to /etc/init.d/rcS after "/bin/mount -t sysfs" :

/sbin/modprobe g_serial

If you'd prefer to have serial and ethernet, and made the kernel config change to enable it as described above, replace g_serial with g_cdc in the line above.

Once you've loaded the g_serial module one way or another and started a getty, you can connect the Kobo to your PC. When prompted for what to do, say "keep reading". You'll discover that the Kobo appears as a USB serial port that you can connect a terminal to. On a Linux host it should be /dev/ttyACMx (/dev/ttyACM0 if you have no other USB serial ports); check dmesg to be sure. You can use a terminal program like minicom, gtkterm or picoterm to talk to the port. The login is "root" and there is no password unless you set one later. You can transfer files using zmodem's rx command if you need to, use the usb ethernet module, or ftp them over wifi.

Unlike telnet over wifi, this serial console stays up no matter what antics nickel performs, and doesn't require the device to be open. It's similarly easy to use g_ether to expose an Ethernet interface or g_cdc to do both ethernet and serial at once. It's just a real pain that using them prevents the g_file_storage module used by the Kobo to manage the onboard memory from working!

Compiling Qt plugins for Kobo Wifi

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

The Kobo Wifi's main user interface application, nickel, is a Qt Embedded app. It drives the ePaper display via a Linux framebuffer interface and some ioctl calls to trigger display refreshes. This is abstracted away from the app behind a QWS (Qt Window System) driver called broadsheet_ioctl that's statically linked into nickel.

Because the QWS driver is only in nickel, we can't really write stand-alone Qt applications to run on the Kobo without reimplementing it. However, the nickel app has a simple plugin interface you can use to load your own code as a shared library.

To do this, you need a full Kobo development environment, including a cross-compiled Qt Embedded build that matches the one used on the Kobo. Once you have that, you can build apps according to the template provided by examples/poker, then drop the shared library produced into /usr/local/Kobo on the device to get it to load.

You now have some totally useless code running on your Kobo. Congratulations. There's no way to invoke it. For now, I'm testing by replacing the poker plugin with my own code, but it's still painful to access it so it's not desirable for real-world use. It may prove necessary to introspect the Qt widget tree or even runtime patch classes to get useful functionality loaded by plugins without having access to the nickel sources.

If your plugin doesn't load, it may be helpful to telnet into your Kobo and create a script like this, called (eg) /usr/local/relaunch-nickel.sh

#!/bin/sh
# Kill off the old nickel
pkill nickel
# Clear plugin cache
rm -f /mnt/onboard/.kobo/Trolltech.conf
# Restart nickel
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib:
export NICKEL_HOME=/mnt/onboard/.kobo
export LD_LIBRARY_PATH=/usr/local/Kobo
export QWS_KEYBOARD=netronix
export INTERFACE=eth0
export LANG=en_US.UTF-8
QT_DEBUG_PLUGINS=1 /usr/local/Kobo/nickel -qws -display broadsheet_ioctl >/mnt/onboard/.kobo/nickel.log 2>&1 &

When executed, this script will restart nickel, causing it to write a debug log to the .kobo directory on the user flash and print detailed information about plugin loading. This might help you figure out what's going on. nickel prints a fair few warnings during normal startup, so don't be thrown by that.

When developing for Qt on Kobo, the Qt for Embedded Linux Environment Variables document may be helpful, though it doesn't mention QT_DEBUG_PLUGINS.

It can also be helpful to determine how Qt was built on your target device. On the Kobo Wifi:

$ strings libQtCore.so | grep 'Build key'
Build key:           arm linux g++-4 full-config

Taking a disk image of the Kobo Wifi without opening the device

This article is part of an extended series on Kobo development and investigation

---

Any attempt to modify your device is risky. Your device might be permanently destroyed. If you cannot risk the permanent loss of your device, so not attempt to modify it. Something can easily go wrong if you do the wrong thing - or follow out of date instructions. I don't continually maintain and test these instructions, so they might well qualify.

If you can't afford to lose it forever, physically open your Kobo up before doing anything, remove the SD card, and take a disk image of it using a PC with a card reader. I use `dd` with a Linux box to make sure I get a complete image of the entire card.

Note that you could break the case or retaining clips when opening your Kobo. I didn't and I opened mine quite a few times, but who knows if they've changed assembly techniques, plastics, etc since then.

Messing with embedded devices that aren't designed to be hacker-friendly is risky. Deal with it, or don't mess with the device.

---

Once you've enabled telnet on your Kobo Wifi, you can use it to transfer a disk image of the Kobo's entire firmware image on the internal MicroSD card. This will give you a backup copy of your Kobo's firmware on your computer.

You can't restore the firmware image without physically opening your Kobo if you mess things up badly enough for it not to boot, but hopefully you won't do that. The Kobo seems to be amazingly hard to break unless you corrupt its file systems, overwrite its kernel, etc. Anyway, this way you can avoid opening the Kobo until/unless you ever need a total firmware restore.

Imaging the Kobo's firmware over wifi

To image the disk of a running system, we need all its file systems in read only mode. This turns out to be trivial to achieve with the Kobo, though it requires terminating the main eReader process ("nickel") so you'll want to reboot after you've done it. During the process, the screen will display whatever was on it before you ran the commands to image the system and none of the keys will respond. Just:

pkill nickel
mount -o ro,remount /mnt/onboard
mount -o ro,remount /
nc -l 9984 < /dev/mmcblk0

Now, take note of the IP address printed by the commands above and use it in this command on your main computer:

nc IP_OF_KOBO 9984 > kobo_internal_microsd.img

The disk imaging process will take a while over wifi, and won't show any progress. Mine took about twenty minutes. If you want progress indication, open a second terminal and run:

watch "du -ms kobo_internal_microsd.img"

... to get a count in megabytes of data transferred so far.

Finally, reboot the kobo to bring it back to normal. You can do this by pressing and holding the power button for 6+ seconds, or by typing control-C then "reboot" and enter into the telnet command line on the kobo.

The image saved on your computer should be 1977614336 bytes for a Kobo Wifi with a 2GB internal card (~1.1GB user accessible memory).

If you have any issues, try a different port number with nc.

^C
reboot

Once you have the Kobo's disk image, you can mount the partitions within it to examine them, and you can extract some individual components of the unpartitioned space in the card.

Imaging the Kobo's firmware to an SD card

In principle you can also image the Kobo's firmware to an SD card instead of using wifi. This will be much easier if you don't run Linux. You need a 2GB or larger card.

To image the Kobo's firmware to an SD card you can telnet in to the kobo and make the file systems read-only as above. Make sure you unmount the external sd card /mnt/sd if it is mounted. Then, instead of getting the IP address and running netcat in listen mode, you can just `dd' the internal mmc card to the external sd card, `sync' to force it to flush, and `reboot'.

Alternately, you may prefer to leave the external SD card mounted, remount everything else read only, then run:

dd if=/dev/mmcblk0 | gzip > /mnt/sd/kobo.img.gz

... to make a compressed image (slow!) on the file system on the SD card. This will be easier to work with under Windows, which doesn't work well with raw devices.

Examining the firmware image

gunzip the image if it's compressed. Either back up a copy of the firmware image that you will not touch, or make the firmware image read-only with chattr +i.

You may now mount the three file systems on the image using this little script - or by hand by using "fdisk -l" to dump the partition table, then calculating the offsets to pass to losetup. If you want to use my script, I make no promises that it won't eat your system and your cat, so be careful. Save the following to "kobomount.sh" and mark it executable:

#!/bin/bash
set -e -u
if test $# -ne 1 ; then
  echo "Usage: $0 firmware.img"
  exit 1
fi
IMG=$1
i=0
sudo -v
for offset in $(sfdisk -d kobo_2gb_microsd_image_v174.img | grep start | cut -d : -f 2 | awk '{print $2}' | sed 's/,//g' |grep -v ^0$); do
  sudo losetup --offset $(( $offset * 512 )) /dev/loop$i "$IMG"
  sudo mkdir -p /mnt/kobo/$i
  sudo mount -o ro /dev/loop$i /mnt/kobo/$i
  ((i++))||true
done

You may then aim it at your firmware image and it'll mount the contained partitions on /mnt/kobo/0 (the recovery partition), /mnt/kobo/1 (the main OS) and /mnt/kobo/2 (the user flash partition). All will be read-only.

You may also want to extract some blobs from the unpartitioned space at the start of the image. That's where the kernel, boot splash image, etc live. I don't have official documentation on the layout of this space, but reading the upgrade scripts in /etc/init.d suggests that these values should be about right. They are UNVERIFIED except for the boot splash image, which I've been able to edit and replace.

IMG=kobo_2gb_microsd_image_v174.img
dd if=$IMG of=serialnumber.bin bs=512 count=1 skip=1
dd if=$IMG of=hwconfig.bin bs=512 count=1 skip=1024 count=2
# The image that ships with the kobo is < 470 512 byte blocks long. It's not clear
# if something else takes the space between the end of the image and the start of
# the epson display binaries, if it's dead space, or if there's room for bigger
# images.
dd if=$IMG of=bootlogo.bmp bs=512 skip=1026 count=470 
dd if=$IMG of=epson_display_setup.bin bs=512 count=1 skip=1920 count=8
dd if=$IMG of=epson_waveform.bin bs=512 count=1 skip=1928 count=$((2048 - 1928))
dd if=$IMG of=kernel.bin bs=512 count=1 skip=2048 count=$((7564-2048))

The Kobo's kernel

We can now examine the existing kernel to learn more about it. This post tells us how to extract the kernel (but it's more easily done with scripts/extract-ikconfig from a kernel tree)

The hard way (which lets us extract the image, not just the config) in brief:

od -A d -t x1 kernel.bin | grep '1f 8b 08 00'
# Note the (decimal) offset, correct for offset of deflate header into line
dd if=kernel.bin bs=1 skip=0013076 | zcat > vmlinux

The easy way to dump the config:

scripts/extract-ikconfig $HOME/kobo/kobofirmwaredump/kernel.img

Boot loading is apparently performed by RedBoot (2.0 according to shipped sources), which should offer tools like `fis list' to report on the structure of the redboot-managed areas of flash storage. Building redboot is a nightmare, so I haven't investigated this yet.

Enabling telnet and ftp access to the Kobo Wifi

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

The Kobo Wifi has this handy 802.11 radio that it really only uses for online shopping. Not only is it calling out to be used for RSS feeds and web browsing, but it is also a really handy tool for developing on the device without having to crack it and solder serial port headers onto the board.

We can gain telnet and ftp access to the device quite trivially; see the pre-made KoboRoot.tgz patch linked to below for the really easy way. (WARNING: The premade patch is for an older Kobo Wifi firmware. See the comments for instructions and newer version info).

If you want to roll your own patch or you're using a different firmware revision, there isn't much to do. The Kobo already has busybox's telnetd, ftpd and inetd on it. All that's necessary is to:

• Edit /etc/inittab and add the lines:

  ::sysinit:/etc/init.d/rcS2
  ::respawn:/usr/sbin/inetd -f /etc/inetd.conf.en
  

• Create /etc/init.d/rcS2 with the content:

  #!/bin/sh
  mkdir -p /dev/pts
  mount -t devpts devpts /dev/pts
  /usr/sbin/inetd /etc/inetd.conf
  

  and run chmod a+x /etc/init.d/rcS2 to flag it executable.
• create /etc/inetd.conf with the content:

  # service_name sock_type proto flags user server_path args
  21 stream  tcp     nowait  root    /bin/busybox ftpd -w -S  /
  23 stream tcp nowait root /bin/busybox telnetd -i
  

• tar our new /etc into KoboRoot.tgz

(Edited for safer method suggested by tjm)

KoboRoot.tgz may then be used to update the device's operating system by putting it in the .kobo directory on the user-accessible flash.

Because it's not designed to be accessible from the outside world, the Kobo doesn't have any root password set. Telnet and FTP access as root will be offered with a blank password by default. You can change that once you telnet in if you like, by typing "passwd" at the root prompt. This won't affect the device's normal operations, only telnet/ftp access.

I created a canned KoboRoot.tgz (for firmware 1.7 ONLY) with these changes to save you the hassle of making them yourself. It was made from the initscripts in firmware release 1.7.4; if you use it with any other firmware and it fails to boot you'll have to do a factory reset. Because there are no binaries in the patch, only a modified /etc/init.d/rcS and a new /etc/inetd.conf, you can easily verify that the code isn't malicious. It has two optional features you can uncomment in /etc/init.d/rcS to (a) syslog to user flash for debugging, and (b) run a user-defined script from .kobo/rc.sh every boot. Both are commented out so they do nothing unless you edit the script to enable them.

Unlike modifying the boot splash screen, this is a pretty safe change as it's completely erased by a factory reset of the device.

Changing the Kobo's boot image

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

The Kobo's boot splash image is stored in unpartitioned space on the internal MicroSD flash. It's an 600x800 4-bit greyscale Windows bitmap (not RLE compressed) starting 1026 512-byte sectors from the beginning of /dev/mmcblk0 (the onboard SD).

Before trying to modify the boot splash image or anything else on the onboard SD card, make a backup of your Kobo's firmware by disk-imaging the internal SD card. If you don't back it up and you damage the kernel or bootloader you won't even be able to factory reset your device; it'll be useless.

If you use a bitmap that is too large you will render your kobo useless, so be careful. I've verified a bitmap of 240120 bytes, saved from Adobe Photoshop CS2, to work on my Kobo Wifi.

`file logo.bmp' should report PC bitmap, Windows 3.x format, 600 x 800 x 4

I haven't found any unix/linux based tools that will write 4-bit greyscale Windows Bitmap images yet, so Photoshop is your best bet. Recommendations appreciated.

The boot splash image may be replaced by putting a compatible bitmap on the user-accessible onboard fat32 partition. It must be located at .kobo/upgrade/logo.bmp. Once placed, disconnect the kobo. It'll detect the upgrade files automatically, apply them, and reboot. On reboot, it'll show the new splash.

(Sorry for the horrible phone-camera pic)

Preparing a development environment for the Kobo Wifi

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

First, I'd like to thank the KoboLabs members for being so helpful and responsive as I try to get up and running with Kobo development. They've pushed several tools, config files, scripts etc to the KoboLabs repository, all of which have drastically reduced the amount of time it's taken to get up and running and largely eliminated the need for reverse-engineering work. This is incredibly unusual for an embedded company, and absolutely wonderful. I was already impressed by their ongoing releases of new firmwares for the original Kobo after the release of the Wifi, as most companies just drop supports old products. Now I'm blown away. Please tell people how damn impressive the Kobo folks are with their after-sales product support and maintenance; I know I will be.

In fact, this whole document has been rendered largely unnecessary by improvements to the KoboLabs code drops. They've posted their build scripts and configurations. To get started, just clone the git repo and read documentation/README.

Overview

You will need to download the CodeSourcery toolchain used for Kobo development, then build the libraries the Kobo uses so that you can link your new executables against them. Qt Embedded, with its dependencies, is particularly crucial. The latest GitHub code drop includes the Qt Embedded configuration used on the Kobo, letting you compile a compatible version of Qt.

Installing Sourcery G++

Download Sourcery G++ 2010q1-202 4.4.1 for your platform and run the installer.

If you're on an x64 RPM-based Linux distro, you may need to install a 32-bit userspace to be able to run the tools. x64 Debian/Ubuntu users will have to install them in the 32-bit chroot, and do all future steps described here within that chroot.

cd ~/Downloads
chmod a+x arm-2010q1-202-arm-none-linux-gnueabi.bin
./arm-2010q1-202-arm-none-linux-gnueabi.bin

Accept the default install path of $HOME/CodeSourcery/Sourcery_G++_Lite when prompted. Accept all other installer defaults.

You now need to symlink your tools to new names, as some of the Kobo makefiles etc expect the tools to be named arm-linux-x where they're named arm-linux-gnuegabi-none-x by CodeSourcery. To do this:

cd ~/CodeSourcery/Sourcery_G++_Lite/bin
for f in arm-none-linux-gnueabi-* ; do ln -s $f arm-linux-${f:23}; done

If all is configured properly, after a logout & login or after running "bash -l" you should now be able to type "arm-linux-g++" and get the response "arm-linux-c++: no input files".

Compiling libraries

Now that we have a toolchain in place, including gcc, binutils, glibc, and all of that provided for us by the wonderful folks at CodeSourcery, we can get on with the real work of preparing our dev environment.

Clone a copy of the KoboLabs git repository to get the required sources and patches:

mkdir ~/kobo
cd ~/kobo
git clone git://github.com/kobolabs/Kobo-Reader.git koboreader

This will take a while, as it's downloading the sources to a lot of libraries the Linux kernel used on the Kobo, and more.

Once you've cloned the repository, it should be easy to get going. A reasonable option might be:

echo "DEVICEROOT=$HOME/kobo/fs" > ~/kobo/build/build-config-user.sh
mkdir ~/kobo/tmp
cd ~/kobo/tmp
../koboreader/build/build-all.sh

... which will use the Kobo build scripts to compile all the libraries and install them in DEVICEROOT, which you set to $HOME/kobo/fs . The unpacked sources and temporary object files will be stored in ~/kobo/tmp , so your git clone of the kobo sources doesn't get messed up by the build.

Now you need to build Qt, which you can do by following the instructions in documentation/README.trolltech

Running your own code on the Kobo

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

The 1.7.4 update tarball (see http://soapyfrogs.blogspot.com/2011/01/getting-kobo-update-urls.html) contains a new copy of /etc/init.d/rcS. Reading it tells us how to replace files on the Kobo's root file system, giving us a way to replace rcS with a patched version that calls out to a shell script on the more easily writable FAT32 main flash, so we can tinker with the system with minimal risk. This initial change is risky, though, as you might render your Kobo useless with a typo!

Replacing files on the Kobo Wifi's root file system

The Kobo Wifi's /etc/init.d/rcS startup script tests for the presence of a number of files within the .kobo subdirectory of the main fat32 partition.

Two notable files are .kobo/Kobo.tgz and .kobo/KoboRoot.tgz. If these files exist, they will be extracted into /usr/local/kobo or / (the root directory) respectively. The system uses this mechanism to apply incremental updates.

We can exploit this mechanism to insert our own files onto the device's main memory or very carefully replace existing files. If you screw it up you might render your Kobo useless and possibly irreparable, so don't try anything unless you don't mind breaking it.

You're much better off copying the firmware from the internal microSD card to a 2G SD card then booting off the external SD by holding "enter" down during power-on. Getting the image for this requires physically opening the Kobo to get to it, unless someone finds an image of it hosted by Kobo somewhere, though.

BTW, the rcS script also looks for .kobo/upgrade and if found, runs /etc/init.d/upgrade-wifi.sh . This is probably the hook for a full upgrade installation. Don't mess with it.

Getting Kobo update URLs

This article is part of an extended series on Kobo development and investigation

Whenever the Kobo eReader connects to wifi, it sends a HTTP/XML request to the Kobo severs with some details about the device, including the current firmware version. The server sends a HTTP/XML response containing various URLs for the Kobo store, etc.

If the firmware version sent by the Kobo isn't the same as the current Kobo firmware, the server also includes the URL for a software update in the HTTP/XML post.

This is apparently only an incremental patch on the main firmware, not a full firmware download, so I still need a full firmware image and will have to extract the MicroSD card from the hardware to get it.

The 1.7.4 update

The URL for the 1.7.4 update is: http://download.kobobooks.com/firmwa...rade-1.7.4.zip

Examination of it tells us some useful things that help open the hardware up some more. See the next post.

How to get the current firmware URL from the server

You can find out the update URL by capturing the wifi traffic of a Kobo with an older firmware when it connects to the network. Use wireshark with the capturing machine's wifi card in monitor mode, or capture the traffic at the router. If you capture off wifi, you'll need to enable 802.11 decryption in Wireshark's protocols->802.11 preferences and enter the key of your network.

In your network capture, look for a request like:

POST /MobileRequest.ashx HTTP/1.1

and in the HTTP/XML response to it you'll find something like:

<UpdateURL>http://download.kobobooks.com/firmwares/kobo2/kobo2-upgrade-1.7.4.zip</UpdateURL>

If you don't have a Kobo with an old firmware, you can achieve the same effect by recording the POST request from an up-to-date Kobo, saving it, modifying it so that <ApplicationVersion>1.7.4</ApplicationVersion> instead reads (eg) <ApplicationVersion>1.6.0</ApplicationVersion>. Because changing the length of the request will corrupt it, you're best off using `sed' to do it. Eg:

sed -e '/^Accept-Encoding/d' -e 's/1.7.4/1.6.0/g' request.txt > request-mod.txt
nc mobile.kobobooks.com < request-mod2.txt > response-mod2.txt

The command deletes the request header that asks the server to gzip the response (for easier reading) and changes the version the "Kobo" announces its self to be running when it connects to the server.

(post series)

Kobo and Kobo Wifi exploration and enhancement

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

I just got myself a new Kobo Wifi, after spending a year drooling over my girlfriend's Kobo 1st gen. I immediately felt the impulse to fiddle come over me, resulting in this collection of information about the hardware, firmware, etc as I research what I can do with it. The result is this series of posts.

There's lots to be learned about it from the sticky threads in the Kobo section of the MobileRead forums. I also strongly recommend reading their wiki article on the Kobo, including the FAQ.

About the Kobo

The Kobo is a great device - it has a few flaws, but may good points that greatly outweigh those flaws. The original Kobo was based on the Netronix EB600, the same board as the Cool-er reader, Astak Mentor Lite, booq, and numerous other devices. Netronix's website is scary-bad, but their products are pretty nice. It's not 100% clear if the Kobo Wifi is still an EB600 product - the kernel refers to the "imx357" - but it's certainly Netronix.

Kobo good points include price, format support, gorgeous eInk screen, light weight and nice feel, and a nice-to-use UI. It's also extremely moddable. It uses a nice standard USB mass storage interface for transferring books so you don't need dedicated vendor crapware applications. With the -wifi you can buy books directly on the device, and with the old version you can use the Adobe Digital Editions app if you don't want to use the Kobo application to buy DRM'd books. Support for DRM-free books is perfect, and is just a matter of dropping them onto the flash.

Bad points include somewhat slow menu navigation (some say page turns too, but I find those just fine), lack of full-justified text display option, useless Bluetooth in the 1st gen, and the inability to remove books from the "currently reading" list. It also has a long pause after unplugging it when you've added books, during which it scans them and adds them to the internal SQLite catalog. It's a pity the Kobo Desktop application can't be used for this while it's connected, though it's wonderful that the device can do it its self and doesn't rely on such an application.

Later firmwares improve some prior issues dramatically, allowing things like charging while reading, and are strongly recommended. The latest firmware is out for both the Kobo and Kobo Wifi; they haven't dropped support for the old generation of hardware, which they deserve real credit for. The use of FAT32 means it risks file system corruption if it's unplugged while transferring books - but this will only happen if you mistreat it, won't cause the device to fail to boot or anything, and is trivial to fix.

The Kobo is massively improved by the use of the free and open source Calibre e-Book management software, which is vastly superior to the Kobo Desktop application in every way. You don't need any management app at all (thanks to USB mass storage support) but Calibre is wonderful for document conversion and cleanup, management, automatic conversion of newsfeeds to ebooks, etc.

Kobo modding - hardware/software info

First: Do not try to reflash your Kobo with firmwares from other EB600 devices. Reports from the mobileread.com forums suggest that this will render your device nonfunctional and make it impossible to restore the original firmwares, though others report having successfully converted a Kobo into a Cool-ER Reader. If you try it, make sure you can afford to replace your eReader or are a JTAG wizard before you try converting it to the Cool-ER firmware or anything like that. At the very least, make sure you take a disk image of its MicroSD card first.

There is much to be learned about the Kobo and many other devices on the MobileRead forums. I'm ringerc there, though I'm not a big forum user so you're better off commenting here or emailing me.

The kobo's kernel, operating system, and user data is stored on an onboard 2GB MicroSD card. There's also a full size externally accessible SD card slot for add-on storage. There are three partitions on the internal microSD - a recovery root FS (ext3), the main root FS (ext3) and the user data partition (fat32). The fat32 partition is exported over USB as a mass storage device when the device is connected in "manage library" mode. The MicroSD card has some unpartitioned space before the recovery partition, where it stores the boot splash screen (800x600x4 Windows BMP), kernel, bootloader, hardware config data, epson display setup data, and probably more.

Because everything is on MicroSD, it's wonderful to hack on. Just pop it open, disk image the MicroSD card using a MicroSD adapter in a PC, and you can recover from pretty much whatever screwup you might make without attacking a flash chip with a soldering iron or learning your JTAG voodoo.

The Kobo (original) apparently presents a root console via its internal 3.3V RS232-like serial port headers, though examination of the Kobo Wifi's board suggests that it may lack these headers. This should allow full access if you're willing to crack your Kobo case - possibly literally, because it's made for easy assembly not easy opening, and is by all accounts a pain to open. I had no real problems opening my Kobo Wifi carefully using a butter knife to pop open all the clips around the edge of the front bezel, but I don't recommend that you try this unless you don't mind the risk of breaking it.

I've been informed by someone working on the Kobo that there is no "trusted boot" with signed kernel and bootloader. The Kobo uses u-boot (the Kobo Wifi may use Redboot; not sure yet) and a regular Linux kernel (2.6.28 for the -wifi). Kernel and u-boot sources, along with some userspace helper apps and other things of interest can be found in the Kobo git repository on GitHub. It's a binary drop of tarballs in git (ugh); there's no version history and no way to differentiate Kobo mods from the original Netronix-supplied kernel etc, but at least it's there. Props to gtalusan for posting the updated sources for the new Kobo Wifi model less than a day after I asked him about them. He's also been wonderful about posting updates, fixes, config files etc as I get further into setting up a Kobo development environment.

The numerous ways of updating the Kobo firmware and root file system

The Kobo has an amazing number of different ways to perform firmware updates, plus several ways to apply small incremental patches to the root file system contents.

Smaller firmware patches are applied by putting a file called KoboRoot.tgz in the .kobo directory of the main onboard FAT32 flash. Upon reboot, this is untarred into the / directory then deleted. Another file named Kobo.tgz (also in .kobo) is untarred into /usr/local/Kobo if found, making for somewhat safer updates when only the Kobo application is being changed. There also seems to be a mechanism to update the Kobo firmware using the .kobo/update file, but I recommend staying away from that.

Firmware updates to the Kobo may be performed by copying a new firmware archive file to a separate piece of FAT32-formatted flash. The device then uses that file to rewrite its firmware. There's no direct flashing over USB to worry about. The Kobo Wifi can download new firmwares over wifi instead of having them dropped on it via USB mass storage. Both the Kobo and Kobo Wifi can also be updated via the Kobo Desktop application or by a simple "cp" of the firmware file by putting the device in firmware upload mode then connecting it over USB.

USB firmware upload mode (which seems to use the minimalist linux recovery partition on the microUSB) is entered by turning the device off and unplugging it from USB, then holding down the Menu key then pressing power. Continue to hold Menu until "Initializing USB partition" appears on the screen. The device will show "Please connect your eReader to USB" on the display. Doing so will mount an empty 200MB FAT32 volume labeled "KoboUpgrade" from a USB mass storage volume from "Netchip Technology" (usb 0525:a4a5). You may safely unmount and unplug the Kobo without copying a firmware file to the partition; it will briefly report that it's "updating software" then reboot back into the old firmware with all your books etc unchanged.

Kobo firmware updates can also be performed via the SD card. Some Canadian users received SD cards with firmware updates for the original Kobo. Reports suggest that it prefers <= 2GB SD cards for updates (not SDHC). Instructions here. I have not tested this. Do not try to install firmwares for other eReaders on your Kobo unless you don't mind rendering it permanently non-functional.

The Kobo can boot off an SD card - just hold down the middle button of the D-PAD ("enter") during power-on, until the first five black boxes of the boot progress bar have filled in. This can be used to test new firmwares without rewriting the onboard flash.

I've yet to build a new firmware for the Kobo, though I've been using the KoboRoot.tgz patch method on mine in an attempt to get telnet and FTP access over wifi. Figuring out how it's put together isn't too hard, but it's unlikely I'll be able to get the rights to distribute new firmwares, so using the Kobo's firmware patch mechanism is likely to be safer and more useful.

The Kobo Wifi runs on an ARMv6 CPU on a Motorola MX35 3-Stack compatible board. It has 128MB of RAM. (I previously said "ARM6" but this is incorrect, see comments).

[root@(none) /]# cat /proc/cpuinfo 
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 530.84
Features        : swp half thumb fastmult vfp edsp java 
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3

Hardware        : Freescale MX35 3-Stack Board
Revision        : 35120
Serial          : 0000000000000000

What runs on the Kobo

The Kobo Wifi runs Linux 2.6.28, glibc 2.11.1, Busybox v1.17.1, and Qt Embedded 2.6.2 among other things. A fuller list can be found in the KoboLabs git repository.

The busybox configuration is fairly complete, though it lacks a bbconfig command. The full list is easily gained by running "busybox" without arguments on the Kobo, so I won't reproduce it here.

The sd8xxx wireless driver is loaded as a module. Everything else seems to be built in to the kernel, but module support is present.

The Kobo Wifi binaries were developed with CodeSourcery Sourcery G++ Lite 2010q1-202 4.4.1. To build plugins for the Kobo Qt application, rebuild Kobo binaries, or build your own, you will need these tools.

PostgreSQL automatic crash dumps for windows

It seems my first real* PostgreSQL patch has been accepted and committed! By the time patch review was done I think it was almost more Magnus Hagander's patch than mine, but I'm still very happy to have put it together and seen it through into Pg mainline. Thanks very much to Marcus for testing, enhancing and committing the patch.

PostgreSQL 9.1 will now have automatic crash dump generation under Windows. That'll allow Windows users to run their production sites with crash dumps so we can do post-mortem debugging without holding up their server or requiring them to run it under a debugger. That gives us a much better chance of tracking down the cause of hard-to reproduce or intermittent faults. To activate the crash dump collection feature, just create a crashdumps directory in the data directory and grant the postgres user (or whatever user your server runs under) "full control" of that folder in Properties->Security. No configuration file changes or server re-starts are needed.

Helpfully, this change will also let the majority who don't know what Visual Studio or windbg.exe are send crash dumps off to someone more experienced with the tools already configured. There should be much less need for this page - and while it took me long enough to write, I'll be glad to see it rendered unnecessary.

* The one-liner fix for the X.509 client certificate validation bug doesn't really count, despite the truly epic amount of convincing required to get the patch applied.