Kobo and Kobo Wifi exploration and enhancement

This article is part of an extended series on Kobo development and investigation

---

Apparently I have to make this more obvious:

• Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
• Any device hacking is risky. If you can't afford to break your device, leave now.

---

I just got myself a new Kobo Wifi, after spending a year drooling over my girlfriend's Kobo 1st gen. I immediately felt the impulse to fiddle come over me, resulting in this collection of information about the hardware, firmware, etc as I research what I can do with it. The result is this series of posts.

There's lots to be learned about it from the sticky threads in the Kobo section of the MobileRead forums. I also strongly recommend reading their wiki article on the Kobo, including the FAQ.

About the Kobo

The Kobo is a great device - it has a few flaws, but may good points that greatly outweigh those flaws. The original Kobo was based on the Netronix EB600, the same board as the Cool-er reader, Astak Mentor Lite, booq, and numerous other devices. Netronix's website is scary-bad, but their products are pretty nice. It's not 100% clear if the Kobo Wifi is still an EB600 product - the kernel refers to the "imx357" - but it's certainly Netronix.

Kobo good points include price, format support, gorgeous eInk screen, light weight and nice feel, and a nice-to-use UI. It's also extremely moddable. It uses a nice standard USB mass storage interface for transferring books so you don't need dedicated vendor crapware applications. With the -wifi you can buy books directly on the device, and with the old version you can use the Adobe Digital Editions app if you don't want to use the Kobo application to buy DRM'd books. Support for DRM-free books is perfect, and is just a matter of dropping them onto the flash.

Bad points include somewhat slow menu navigation (some say page turns too, but I find those just fine), lack of full-justified text display option, useless Bluetooth in the 1st gen, and the inability to remove books from the "currently reading" list. It also has a long pause after unplugging it when you've added books, during which it scans them and adds them to the internal SQLite catalog. It's a pity the Kobo Desktop application can't be used for this while it's connected, though it's wonderful that the device can do it its self and doesn't rely on such an application.

Later firmwares improve some prior issues dramatically, allowing things like charging while reading, and are strongly recommended. The latest firmware is out for both the Kobo and Kobo Wifi; they haven't dropped support for the old generation of hardware, which they deserve real credit for. The use of FAT32 means it risks file system corruption if it's unplugged while transferring books - but this will only happen if you mistreat it, won't cause the device to fail to boot or anything, and is trivial to fix.

The Kobo is massively improved by the use of the free and open source Calibre e-Book management software, which is vastly superior to the Kobo Desktop application in every way. You don't need any management app at all (thanks to USB mass storage support) but Calibre is wonderful for document conversion and cleanup, management, automatic conversion of newsfeeds to ebooks, etc.

Kobo modding - hardware/software info

First: Do not try to reflash your Kobo with firmwares from other EB600 devices. Reports from the mobileread.com forums suggest that this will render your device nonfunctional and make it impossible to restore the original firmwares, though others report having successfully converted a Kobo into a Cool-ER Reader. If you try it, make sure you can afford to replace your eReader or are a JTAG wizard before you try converting it to the Cool-ER firmware or anything like that. At the very least, make sure you take a disk image of its MicroSD card first.

There is much to be learned about the Kobo and many other devices on the MobileRead forums. I'm ringerc there, though I'm not a big forum user so you're better off commenting here or emailing me.

The kobo's kernel, operating system, and user data is stored on an onboard 2GB MicroSD card. There's also a full size externally accessible SD card slot for add-on storage. There are three partitions on the internal microSD - a recovery root FS (ext3), the main root FS (ext3) and the user data partition (fat32). The fat32 partition is exported over USB as a mass storage device when the device is connected in "manage library" mode. The MicroSD card has some unpartitioned space before the recovery partition, where it stores the boot splash screen (800x600x4 Windows BMP), kernel, bootloader, hardware config data, epson display setup data, and probably more.

Because everything is on MicroSD, it's wonderful to hack on. Just pop it open, disk image the MicroSD card using a MicroSD adapter in a PC, and you can recover from pretty much whatever screwup you might make without attacking a flash chip with a soldering iron or learning your JTAG voodoo.

The Kobo (original) apparently presents a root console via its internal 3.3V RS232-like serial port headers, though examination of the Kobo Wifi's board suggests that it may lack these headers. This should allow full access if you're willing to crack your Kobo case - possibly literally, because it's made for easy assembly not easy opening, and is by all accounts a pain to open. I had no real problems opening my Kobo Wifi carefully using a butter knife to pop open all the clips around the edge of the front bezel, but I don't recommend that you try this unless you don't mind the risk of breaking it.

I've been informed by someone working on the Kobo that there is no "trusted boot" with signed kernel and bootloader. The Kobo uses u-boot (the Kobo Wifi may use Redboot; not sure yet) and a regular Linux kernel (2.6.28 for the -wifi). Kernel and u-boot sources, along with some userspace helper apps and other things of interest can be found in the Kobo git repository on GitHub. It's a binary drop of tarballs in git (ugh); there's no version history and no way to differentiate Kobo mods from the original Netronix-supplied kernel etc, but at least it's there. Props to gtalusan for posting the updated sources for the new Kobo Wifi model less than a day after I asked him about them. He's also been wonderful about posting updates, fixes, config files etc as I get further into setting up a Kobo development environment.

The numerous ways of updating the Kobo firmware and root file system

The Kobo has an amazing number of different ways to perform firmware updates, plus several ways to apply small incremental patches to the root file system contents.

Smaller firmware patches are applied by putting a file called KoboRoot.tgz in the .kobo directory of the main onboard FAT32 flash. Upon reboot, this is untarred into the / directory then deleted. Another file named Kobo.tgz (also in .kobo) is untarred into /usr/local/Kobo if found, making for somewhat safer updates when only the Kobo application is being changed. There also seems to be a mechanism to update the Kobo firmware using the .kobo/update file, but I recommend staying away from that.

Firmware updates to the Kobo may be performed by copying a new firmware archive file to a separate piece of FAT32-formatted flash. The device then uses that file to rewrite its firmware. There's no direct flashing over USB to worry about. The Kobo Wifi can download new firmwares over wifi instead of having them dropped on it via USB mass storage. Both the Kobo and Kobo Wifi can also be updated via the Kobo Desktop application or by a simple "cp" of the firmware file by putting the device in firmware upload mode then connecting it over USB.

USB firmware upload mode (which seems to use the minimalist linux recovery partition on the microUSB) is entered by turning the device off and unplugging it from USB, then holding down the Menu key then pressing power. Continue to hold Menu until "Initializing USB partition" appears on the screen. The device will show "Please connect your eReader to USB" on the display. Doing so will mount an empty 200MB FAT32 volume labeled "KoboUpgrade" from a USB mass storage volume from "Netchip Technology" (usb 0525:a4a5). You may safely unmount and unplug the Kobo without copying a firmware file to the partition; it will briefly report that it's "updating software" then reboot back into the old firmware with all your books etc unchanged.

Kobo firmware updates can also be performed via the SD card. Some Canadian users received SD cards with firmware updates for the original Kobo. Reports suggest that it prefers <= 2GB SD cards for updates (not SDHC). Instructions here. I have not tested this. Do not try to install firmwares for other eReaders on your Kobo unless you don't mind rendering it permanently non-functional.

The Kobo can boot off an SD card - just hold down the middle button of the D-PAD ("enter") during power-on, until the first five black boxes of the boot progress bar have filled in. This can be used to test new firmwares without rewriting the onboard flash.

I've yet to build a new firmware for the Kobo, though I've been using the KoboRoot.tgz patch method on mine in an attempt to get telnet and FTP access over wifi. Figuring out how it's put together isn't too hard, but it's unlikely I'll be able to get the rights to distribute new firmwares, so using the Kobo's firmware patch mechanism is likely to be safer and more useful.

The Kobo Wifi runs on an ARMv6 CPU on a Motorola MX35 3-Stack compatible board. It has 128MB of RAM. (I previously said "ARM6" but this is incorrect, see comments).

[root@(none) /]# cat /proc/cpuinfo 
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 530.84
Features        : swp half thumb fastmult vfp edsp java 
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3

Hardware        : Freescale MX35 3-Stack Board
Revision        : 35120
Serial          : 0000000000000000

What runs on the Kobo

The Kobo Wifi runs Linux 2.6.28, glibc 2.11.1, Busybox v1.17.1, and Qt Embedded 2.6.2 among other things. A fuller list can be found in the KoboLabs git repository.

The busybox configuration is fairly complete, though it lacks a bbconfig command. The full list is easily gained by running "busybox" without arguments on the Kobo, so I won't reproduce it here.

The sd8xxx wireless driver is loaded as a module. Everything else seems to be built in to the kernel, but module support is present.

The Kobo Wifi binaries were developed with CodeSourcery Sourcery G++ Lite 2010q1-202 4.4.1. To build plugins for the Kobo Qt application, rebuild Kobo binaries, or build your own, you will need these tools.

PostgreSQL automatic crash dumps for windows

It seems my first real* PostgreSQL patch has been accepted and committed! By the time patch review was done I think it was almost more Magnus Hagander's patch than mine, but I'm still very happy to have put it together and seen it through into Pg mainline. Thanks very much to Marcus for testing, enhancing and committing the patch.

PostgreSQL 9.1 will now have automatic crash dump generation under Windows. That'll allow Windows users to run their production sites with crash dumps so we can do post-mortem debugging without holding up their server or requiring them to run it under a debugger. That gives us a much better chance of tracking down the cause of hard-to reproduce or intermittent faults. To activate the crash dump collection feature, just create a crashdumps directory in the data directory and grant the postgres user (or whatever user your server runs under) "full control" of that folder in Properties->Security. No configuration file changes or server re-starts are needed.

Helpfully, this change will also let the majority who don't know what Visual Studio or windbg.exe are send crash dumps off to someone more experienced with the tools already configured. There should be much less need for this page - and while it took me long enough to write, I'll be glad to see it rendered unnecessary.

* The one-liner fix for the X.509 client certificate validation bug doesn't really count, despite the truly epic amount of convincing required to get the patch applied.

Kogan KGN1080P32VAA - Avoid this TV and any other with a CultraView CV119MA mainboard

The Kogan KGN1080P32VAA TV is a lemon. Avoid it, and Kogan.

Support

Before I go on, I want to note that Kogan's support were responsive, if not helpful, about these issues. They acknowledged the EDID defect, and while they won't fix it or anything else that's wrong with the TV they did credit me $40 for my trouble, showing some sign they accept their error. Kogan have been much better than, say, Belkin when it comes to supporting their products. Given that they still haven't actually fixed anything, that says scary things about the industry, doesn't it? Anyway, here's why you should avoid this product, anything else by CultraView, and possibly anything else by Kogan:

720p EDID on a 1080p panel, mainboard programmed wrong

When I bought the KGN1080P32VAA I expected (and got) a 1080p TV with cheap and nasty LCD panel; this is, after all, a $600 1080p TV. Fine, you get what you pay for, and it's still a 1080p full HD panel.

What I didn't expect was an incorrectly programmed mainboard that sends the EDID for a 720p TV rather than a 1080p TV, making the TV not "full hd" at all (and in fact worse than native 720p due to scaling) for anything that respects the EDID! What does this say about quality control and product testing? I certainly didn't expect Kogan to have no interest in issuing a firmware update to fix it, given that the TV's mainboard, by physical examination determined to be the CultraView CV119MA is still in production and supports firmware updates via USB key. CultraView don't respond to queries and don't offer their tools or firmware for download without a username and password to their vendor/partner FTP site. Kogan say:

We have tried to get a firmware upgrade for you TV however we cannot get solution for this issue.

I would still suggest you to use the VGA as you get Full HD resolution on VGA without any issue.

Alas, there's only one VGA input and associated 3.5mm audio port. I didn't buy a TV with only one (working) input. Additionally, at 1920x1080 the (analog) VGA looks ... kind of nasty. This is partly explained by the picture issue described later.

HDMI audio and overscan

The problems with HDMI don't end with the EDID, though. The CultraView CV119MA, as shipped by Kogan, turns on overscan on digital inputs when it detects a HDMI audio signal. No HDMI audio, no overscan. HDMI audio, overscan. This means you can't get decent picture quality and HDMI audio at the same time - you have to endure the image butchery of overscan and overscan compensation, or have no audio on that input. Overscan makes no sense for digital inputs, where a perfect 1:1 pixel representation is sent and there are no scan line fringes, etc, to deal with. Sadly overscan does need to be an option because some older/broken digital output devices produce images with overscan for compatibility with ancient TVs, but should never be the default, and is something you must be able to disable. It certainly shouldn't be controlled by whether or not HDMI audio is being sent!

It's possible to disable overscan by disabling HDMI audio - if your device supports this. For my media PC, that requires telling the nVidia driver to completely ignore the EDID so it doesn't detect the HDMI audio capability of the output. I had to override the EDID anyway because the TV sends a totally bogus EDID with only 1080i and 720p resolutions over HDMI. There are a couple of ways to override the EDID though - I found that overriding just the resolutions left me with an overscanned 1080p image, while overriding the whole EDID got me a 1:1 pixel sharp-ish 1080p image. The difference: HDMI audio.

The TV doesn't offer a "1:1 pixel" mode in its zoom/aspect list, nor in its menus. The only way to control overscan is via HDMI audio. I'm gobsmacked.

Vendors unwilling to fix anything

Neither Kogan nor CultraView will supply a firmware update, or the tools to create one using the vendor customisation tools CultraView offer to vendors. Neither will supply the documentation for the mainboard, which has "PC DEBUG" (JTAG?) and "PC INPUT" (DB9 serial headers? Maybe?) on it and has updatable firmware. Examination of a firmware image I found on the 'net confirms that it's capable of being reflashed via USB.

Brightness controls pixel values not backlight

The problems don't end with a misprogrammed EDID and idiotic overscanning behaviour there, though. When you change the "brightness" setting in the TV, it doesn't adjust the backlight, it scales the pixel values down toward the black end, so you lose contrast and get horrid banding. While this can be written off as a "cheap TV" problem, it's one I never even imagined might be possible.

Horizontal blur at native 1080p

Worse: Even now that I've finally achieved a non-overscanned image at the panel's native 1080p resolution, the image is still crap. By creating a few stippled and striped b&w test patterns, I was able to determine that horizontal lines are sharp and clear - so there's no vertical scaling or distortion. An array of 1-pixel wide b&w vertical lines, though, are blurred and smudged when displayed on the TV, as if they've been scaled down then up, or up then down, before display.

The test image, unscaled, appears below. If your LCD display is set to native resolution and your browser isn't scaling the image, this image should appear as a block of fine vertical lines.

... and here's a photo of what you see on the Kogan TV. The connected machine was running Linux, but I see the same problem under Windows 7's (to use Media Center) over HDMI:

Here's an image I scaled down by 5%, then back up to original size, which demonstrates a similar effect:

Imagine what this does to text, typefaces, and most kinds of pattern. Argh!

It's just too crap

Before you say "It's a TV, not a computer monitor", let me quote from Kogan's product page for their new model 32" 1080p HD TV, the 1080P-BD32:

32"/ 81cm Full HD panel (1920x1080)
Full High Definition panel ensuring crystal clear TV with Progressive Scan up to 1080P. Doubles up as an excellent computer monitor.

While that text did not appear on the product page for my TV, it shows that Kogan, like me, think that it's reasonable to use modern HDTVs as monitors. In any case, I'm using mine for a lounge room media PC, not a regular PC, and just want it to work! I don't expect amazing quality, but actually displaying the pixels I send it and doing so with working HDMI audio surely isn't too much to ask?

After hours of work and several conversations with Kogan, I finally have the TV running at its rated 1920x1080 (1080p) resolution without horribly butchering the image with overscan. Of course, to get that far I had to override the EDID and disable HDMI audio. If there's no HDMI audio signal the TV plays the 3.5mm input on all HDMI channels - but that means you have only one input for all your channels, making the HDMI combined audio/video feature spectacularly useless, and rendering the TV effectively capable of only one distinct A+V input among from the 2 HDMI and 1 VGA ports. Needless to say, this sucks.

At this point the TV is annoying me enough that I'm planning on gutting my old laptop and using its 1080p capable LVDS output to drive the TV's panel directly, bypassing the TV mainboard. That way I'll be able to see how much of this is the panel and how much is that CultraView mainboard. If the panel is OK, I'll grab a little ARM or Via Nano board with a couple of HDMI inputs and a high-res LVDS output and build that into the case as an embedded media PC, getting rid of the CultraView part entirely.

Well, I've certainly learned one thing. I won't be buying from Kogan again unless I can test the product in person with my equipment before buying. They assemble products from 3rd party parts - which I knew. They don't push issues back to those suppliers and get fixes for them, which I didn't know and didn't expect. Poor show, Kogan.

Are you a programmer+sysadmin+support person at a small company? You need to read this.

I'm a sysadmin, developer of in-house software, and end-user tech support person at a small company. I wear a lot of hats, and do everything from Java EE applications to data recovery from crashed user laptops. If you can't open that email attachment or get to that website, I'm your man.

Needless to say, this makes the programming side rather challenging. I've always thought of interruption as the enemy of effective programming, and I've observed my effectiveness falling as the rate of interruption increases.

This article gives me reason to reconsider that idea, and consider trying to plan for and work with interruption instead. I cannot recommend it enough.

http://www.stevestreeting.com/2010/09/04/work-2-0/

On SSDs

I suspect that some of the ideas behind the design of SSD drives as currently sold are rather flawed. Update: See the end of the article for an alternative viewpoint, though.

These drives are embedded computers, with their own CPUs, RAM, firmware, etc. They are much more complicated than they should be for a simple storage device, and are more like mini RAID controllers than they are like hard disks.

Notes on gdb use

I find myself looking up how to do certain things with gdb, because I use it rarely enough not to remember, but frequently enough for these to be somewhat annoying. This post notes how to handle things like signals and paging in gdb.

BoB (Belkin F1PI243EGau) DNS is still broken

Update Oct 19, 2010:Contacting Belkin sales and customer feedback and pointing out my increasing efforts to show how poorly they've performed in public finally got a response from Belkin. Admittedly the response is to say they've put the issue though to an "overseas engineer" ... but maybe something will happen. More likely, it'll stay trapped in another layer of disinterest and poor management, this time one I can't apply direct pressure to.

UPDATE Oct 6, 2010:An indirect approach, by bothering a friend who works at iiNet, got this issue through the support wall and to people who can deal with it. Belkin has been notified at a higher level too. It's a real shame that a clearly demonstrable issue like this got stuck behind support people at both companies, to the point where I had to bother a friend who shouldn't have to deal with this stuff just to get the issue looked into. Sometimes tech support acts as a barrier that prevents a company from finding out about real problems, an issue I've seen not only with iiNet and Belkin but with endless other companies. Anyway, hopefully Belkin will be getting onto this now.

---

A year ago, I reported to iiNet that AAAA lookups in the BoB (F1PI243EGau) DNS forwarder were always timing out, rather than returning SERVFAIL or correctly forwarding the query to the upstream server. This is the cause of the slow browsing issues reported for the BoB.

A couple of months ago I got hold of a pre-release firmware that fixed this, and about a month ago the firmware was finally put up for public use. This fixes the AAAA issues, so browsers like Firefox and Safari on IPv6-capable operating systems like Mac OS X and Windows 7 don't take an eternity to resolve every DNS query.

Unfortunately, Belkin didn't take this as a hint to properly test their DNS forwarder. They fixed AAAA lookup, but didn't fix it to return SERVFAIL when it encountered something it didn't understand, and failed to test other record types like TXT and SRV.

Sure enough, TXT and SRV lookup have the same problem. This is currently causing problems with Google Talk (using Pidgin) that require the configuration of a fallback connect server to bypass TXT record lookup.

Belkin support do not understand the problem. The low-level support folks at iiNet don't seem to get it either. Neither are passing the problem on to somebody with the experience and knowledge to understand the problem, and neither seem to have access to suitable hardware - or the inclination to use it if they do - to verify the issue.

Here's the explanation I sent to them.