This article is part of an extended series on Kobo development and investigation
Apparently I have to make this more obvious:
- Take a backup of your Kobo's SD card before doing anything. If something doesn't work, you can't fix your Kobo without this backup. I will not send you a backup.
- Any device hacking is risky. If you can't afford to break your device, leave now.
(This post is part of a series on the internals of the Kobo and Kobo Wifi eReaders. The rest of the posts are technically focused; this one was just prompted by increasing frustration with the device.)
I really like the Kobo. It has great hardware, a great price, and some pretty nice software running on it. Unfortunately, it also has some annoying bugs and limitations - many of which wouldn't be too bad,if only there was any sign they'd ever get fixed/resolved. some but not all of which have now been fixed in the 1.9 firmware.
I feel a little bad paying the Kobo out about these issues, because the Kobo folks were really great to deal with and really responsive early on. I've been absolutely blown away by how far beyond narrow-interpretation GPL requirements they've gone with their source releases (including full build scripts and config files) and how quick they were to update their releases when asked about them. Please keep in mind that overall the Kobo is an excellent eReader and the Kobo guys deserve our support for being one of the depressingly few companies who fully accept, understand and act on the obligations that fall on them in exchange for the free use of GPL-licensed software.
(You can't) Go To Page (fixed in 1.9)
There's no way to jump to a page in a book by entering the page number, though the Kobo Wifi uses a handy on-screen keyboard for the online store that'd be ideal for the job. The inability to jump straight to page wouldn't be so bad, except that...
Reading position loss sucks
... it loses your page when it runs out of power. It tends to run out of power suddenly and without warning, which doesn't help. Indications are that `nickel', the reader app, doesn't checkpoint your reading position to the SQLite database periodically, it only does it on explicit shutdown or when you go to the home screen.
Factor in the fact that you can't jump straight to a page number, and you discover just how tedious it is to press "next page" enough times to get back to where you were. It's even worse in a book with long chapters, or with no chapter structure at all.
Which of the set I just added scrambled its brains?
How many files are really on this Kobo? Tip: more than "no books found".
(This issue updated 28 Feb to reflect new information; see comments)
At one point book I added to the unmodified orignal Kobo I had at that point caused it to crash during startup and reboot, failing to add that book to the database or recognise any new content added after that book was added*. It rebooted normally into the usual `nickel' reading screen, but didn't see any of the new content. As I'd added a fair few books in that one update, I had no idea which book was the problem and no way to find out. I had to do a laborious binary search where I add/remove half the books at a time, slowly narrowing down the culprit. Tedious beyond belief, especially when it turns out you have more than one book that triggers a bug in Kobo's epub parsing.
To fix this, Nickel needs to write a checkpoint file indicating which book it's about to process, then remove it after successfully processing that book. If it starts up and discovers a checkpoint file in place, it knows it crashed while processing that book and should not try to process it again; instead it carries on with the next book. That way, the Kobo could tell you which books it failed to process, and let you read the rest of the ones you just added.
(Update: I recently had what I thought was another case of this, but even after a factory reset it still couldn't find any books, even previously working ones. A restore from a backup copy of my SD card fixed it. On balance, it's most likely that I broke something subtly ... somehow. The rest of the issues described here affect a stock Kobo, though, with no modifications.)
I'm not currently reading that (fixed in 1.9)
The Kobo keeps track of a "currently reading" list. This rapidly grows to include a long list of books you've looked at, decided you don't want to read right now, and left. You can't remove them from this list without clearing the SQLite database completely or deleting the book off the Kobo. A cosmetic issue, but it becomes an irritation to have many, many "1% read" books in your collection.
Dammit, which is the next book?
You know how series are very popular with authors and publishers these days? They bring in a lot more money, because people are willing to pay more for three (or five, or eight, or the godamm wheel of endless time ogod-make-it-stop-i-beg-you) books than just one huge one? And, frankly, they can be carted around in pieces rather than making you tote around something the size of the Necronomicon that could double as a potent self-defense weapon?
Well, there are lots of series out there. And the Kobo has absolutely no idea that they exist. Each book is individual and complete in and of its self to the Kobo, which doesn't seem to think that you REALLY want to read that six book series in order. I hope you have a smartphone to look up the Wikipedia entry on the series and find out which book is next, or you'll be doing plenty of guesswork.
Most ePub novels contain series and other metadata, but the Kobo doesn't use most of it.
Quote me once
The quotes are a cool idea. They're also hard-coded into Nickel, and there's only a small list of them. After a while, they may begin to drive you just a little bit batty. Why not put them in an XML file that's loaded at launch or on first use, so a bit more variety becomes possible? Or let us hide them? This doesn't bug me much, but it drives my girlfriend (who has a gen1 kobo) up the wall.
Don't just whine about it, do something!
I'd love to fix some of these. I'm a competent C++/Qt programmer, and I already have an ARM development environment set up for the Kobo that lets me compile binaries for the Kobo, including plugins for `nickel' (the Qt-based eReader app). Unfortunately, the KoboLabs GitHub repo doesn't contain any headers for Nickel and the plugin API its self is largely useless without them. Even with a usable plugin API, it's not clear that I could fix or improve many of these without the full sources to Nickel. So, really, all I can do is whine about it and hope the Kobo guys will get around to fixing some of these issues.
*Only once (the second time I thought it'd happened turned out to be a different problem) but it was very frustrating to track the issue down.
Hey Craig. Much appreciate your tech kobo posts - saved me loads of time in getting my development environment up.
ReplyDeleteI suspect your DB or filesystem may be corrupt. I'm a heavy user of Baen and have dozens of their free library ePub's on my kobo. All work just fine. Hope you get it sorted!
Cheers.
3d0g: Thanks for confirming that Baen's books aren't the issue. It seemed unlikely, especially once I saw that their Producer: field is "Calibre", but it seemed initially like that was it. It wasn't, as it turns out: after a factory reset, _no_ books are loading.
ReplyDeleteA factory reset definitely erased my new init script. `nickel' no longer writes its log to the sd card, and the Kobo isn't accessible over telnet or ftp anymore. One of the reasons I've been willing to make small tweaks is that it's so easy to undo. Yet I cannot help but suspect that something I've done has - somehow - caused this one.
No titles at all? I've seen that once... if it stays like this even after a reboot, this may be a damaged *.sqlite database. In this case, attempts to read the database using the Firefox SQLite plugin or the SQLite command-prompt apps on the desktop are likely to also fail.
ReplyDeleteA few possible fixes:
1) (blatantly stolen and adapted from www.bvcaddons.com/FixSqliteDatabase):
$ echo .dump | sqlite3 KoboReader.sqlite > KoboReader.sql
(this saves what can be salvaged from the current, damaged database to a text file)
$ ren KoboReader.sqlite KoboReader.sqlite.bak
$ sqlite3 -init KoboReader.sql KoboReader.sqlite
(this moves the existing problem database out of the way and creates a new one with the same data)
If that works, great. If not, two remaining options (in increasing order of severity) are:
2) Rename the KoboReader.sqlite file and then reboot the device. If the file cannot be found, a new blank database will be created.
3) If all else fails, a factory reset will restore the original database (with the 100 preloaded books and little else) but at the expense of clobbering the entire FAT32 partition. If it comes to this, be sure to make a copy of the *entire* device first as this wipes just about everything.
Did any of your changes affect anything on either of the two Linux-native partitions on the microSD card?
ReplyDeleteIt would seem unlikely to be an *.epub or SQLite database issue at this point if a factory reset would have discarded everything on FAT32 and replaced it with the recovery image.
carlb: Thanks for the tip re database recovery. I've never bothered in the past, though; I just nuke the whole .kobo dir and let the Kobo restore a fresh one from compressed copy it keeps the ext3 partition. I tend to treat my Kobo db as pretty disposable.
ReplyDeleteFWIW, except for the most recent issue with no new books showing up even _after_ a factory reset, I've had all these issues with an unmodified Kobo.
Yes, some of my changes did affect the ext3 partition, but only the user partition not the recovery partition. Nonetheless, I can only imagine that somehow I broke something. The alternative would be low level file system corruption on the user ext3 partition, and I didn't find any evidence of that.
I was able to restore normal functionality by opening the Kobo up, removing the SD card, and `dd'ing a backup I took earlier onto it. Either the restore of the backup, or the fact that I had to disconnect the battery to remove the card, seems to have fixed the issue. I'm still not clear why this failure appeared to be triggered by adding a book - since I'd been adding books every few days, and I hadn't made any changes to the Kobo's FS for weeks. Anyway, a restore from backup fixed it.
Separately to this most recent issue, I* have had a single corrupt ePub crash nickel during indexing, causing the Kobo to fail to see the new book or any other new books until the corrupt book is identified and deleted. OK (if annoying) if you added only one book; really, really painful if you just added 50. It'd be great if Nickel checkpointed its progress to help catch this sort of issue, as I spent a good hour adding and removing chunks of books to isolate the faulty one.
* well, it was my girlfriend's unmodified original Kobo, but close enough
I have to wonder if this firmware is resetting on a watchdog timer of some sort every time it sees something it doesn't like. I'm currently looking at a wi-fi Kobo which shows at least two hundred fewer books in the title list than were loaded from the PC; it seems to get about 10% through the eternally-slow "please wait while your Kobo eReader processes nuked content" and then reboot into the main nickel app. Try to read a book which contains something it can't handle (like the huge HTML tables listing Wikipedia articles) and it reboots there too. A restart with no errors logged seems almost like a device stuck in a rustuck in a rustuck in a rut...*resets* "Please wait while your Kobo eReader takes a snooze".
ReplyDeleteError detection and recovery seems to be all-but-nonexistent in this code; the only error message I've seen on the Kobo screen is the bogus DRM "file is locked" when an *.epub has been manually deleted without removing the corresponding database entries. Anything else that goes wrong, it says nothing and often reboots.
Lets hope the Kobo folks are listening. Fixing the bugs/missing features would be my first wish, but allowing some 3rd party development would be great!
ReplyDeleteI hit up Serbinis way back in June about opening up the API for 3rd party development. He said they'd love to but they were way too small to support the API anytime soon.
ReplyDeletecarlb: The kernel config suggests that there's a watchdog, but I'm pretty sure it's only kernel<->hardware, with no userspace component. I can't see how a malfunction in `nickel' would cause the kernel to stop signalling "alive" to the watchdog periodically.
ReplyDelete3d0g: Honestly, there *isn't* a plugin API, there's just a simple shared library loader and a single hook that's only run by invoking a hard-to-reach easter egg in the UI. There aren't any useful entry points or hooks to *do* anything. Because it's a Qt application you can obtain the static QApplication reference and introspect `nickel' via it to some extent, but it's really difficult to know what you can do w/o having the headers for Nickel. You certainly can't replace instances of classes with extended versions, because you don't have the headers for the original in order to inherit from and extend it. Even then, I doubt Nickel was written for extensibility, so it probably has lots of inline and non-virtual methods in its classes, making it even harder to override behaviour without the ability to recompile nickel its self.
I've pretty much given up trying to use the existing so-called plugin api as a waste of time. If they want 3rd party help (which there's less and less sign of lately) they'll have to open up Nickel under some kind of source-available terms, though not necessarily free to redistribute or anything. It'd be way more engineering effort to make a useful plugin API than it would be to implement most of what people want to do in plugins directly, so I don't see it as worth their while to actually build a real plugin api.
I haven't had the nerve to "hack" my wife's fairly new Kobo WiFi (because if I "break" it I will be in trouble), so I will ask questions.
ReplyDeleteI did download the upgrade firmware to look at the structure to give me a feeling how the device works, although I know it might not be a complete set.
What would happen if nickel was to segfault or fail? Would this make the device reboot? Restart nickel? Hang? I see some discussion of restarting nickel in the compiling qt plugins post, but now what happens if it simply dies. I suppose with telnet access it would be simple to figure out with a quick kill command.
I figure someone may have noticed but on the update site there is a firmware 1.7.5 (Dec 17/2010). Looking at the files in there the only obvious changes are in libnickel (which is the most important part of course). But I only see references to 1.7.4 on the net, and the device only updates automatically to 1.7.4 over WiFi.
As for the lack of bug fixes I have a feeling Kobo is placing most of their attention on the non-Kobo hardware development IE the Android/Iphone/etc software.
I also have this sinking feeling it is possible they lost the original nickel coder, and so updates to the Kobo hardware platform have been disrupted because of this. Although some here may know the developers well enough to know this.
bjb: if nickel crashes, the device's UI stops responding. There's no user-level watchdog to automatically restart nickel, so it'll just sit there with whatever was last on the screen. Pressing and holding the power button to force power-off then turning it back on gets it back to normal.
ReplyDeleteThe reason they don't have a watchdog restarting nickel is probably that (a) it's extremely stable, and (b) it's automatically killed by the udev scripts that're run when USB is plugged in, and shouldn't be restarted until USB is unplugged.
I didn't know about the 1.7.5 firmware. Strange that it's not being pushed. Maybe a fix they specifically send to people having a particular problem?
BTW, I was accosted on the train yet again by someone interested in my Kobo. It seems that the fact that it's white makes it jump out at people more. I blame Apple.
ReplyDeleteAnyway, the lady was REALLY interested in it, having heard of eReaders in general but only being specifically familar with the Kindle with its Amazon lock-in and high price. I was still happy to recommend it to her, as I think it's well worth the money for the amazing convenience and functionality, and it's less problematic than a Kindle.
I'm getting more and more pissed off about the lack of communication about bugs/fixes and any sort of development ETA, though. Going dead-silent is not reassuring or helpful.
Carlb's comment about dumping the database and recreating it is close. I have worked on two malformed databases recently and I have a script that removes the rows causing the issue from the database dump.
ReplyDeleteI have been also working on some scripts to check the database integrity after it is rebuilt.
I have an initial post at:
http://timlegge.blogspot.com/2011/03/fixing-kobos-malformed-database.html
On the MobileRead site there is a mention of a beta test of version 1.9 for the Kobo WiFi, so maybe some relief is on the way!!
ReplyDeleteAnyone know how to become a beta tester? There is a beta signup page on the Kobo blog but it doesn't seem to work (the post goes to a 404 page).
@bjb: 1.9 adds "go to page" and "go to chapter". Big improvement right there. It also adds the ability to remove things from the currently readin glist.
ReplyDeleteThe new firmware has various other improvements that don't cover things mentioned here.
I'd love to pass it on, but I don't have an explicit OK from Kobo to pass on the release candidate download URLs and I don't feel comfortable assuming it's fine without their confirmation. Hopefully they'll push the final firmware soon or respond to your request. Try PMing them on MobileRead.
Hello Craig,
ReplyDeleteI hope you can help me.
I have developed technical content as a large pdf file and I have to send it for further proof reading/evaluation to a number of people on an ebook reader. I was planning to load on a number of KOBO to send it but I need a way to prevent copying of my content by others. One possible way is to use a device ID to lock in to each Kobo.
I am also willing to pay for your efforts.
Please reply soon to fsopari@gmail.com
Thanks in advance
Farooq: You'd have to use Adobe Digital Editions. Contact Adobe for pricing and details. I don't do DRM.
ReplyDeleteFor what it's worth, I think you're wasting your time. If they can't copy the file, they can still take photos of the screen. Hell, you can even scan a Kobo screen with a flatbed scanner!
If they can read it, they can copy it. The law (copyright, trademark and patent law as appropriate) protects you much better than half-assed technical features.
Gidday Craig
ReplyDeleteI know you've abandonded Kobo, but would appreciate a quick opinion.
I've inherited a Bluetooth Kobo. Wondering if I can re-pupose it for something else. I want it to cycle through different display images (say pdfs). Sleep in between.
I am not very experienced with Linux.
Does the above sound possible?
Can I update files to use via the Bluetooth interface?
Any comment appreciated.
Thanks Leigh in NZ
Leigh: Because Kobo have declined to release the epson eInk display driver for the QT Embedded Window System (QWS), it isn't really possible to do this with a Qt app the way you might want to. Even if they had released the QWS driver you'd still have to be capable of doing the required C++ programming with the Qt toolkit to write the program to do what you wanted, and have to be able to then load that program onto the Kobo. Neither is trival.
ReplyDeleteThere is a simple utility on the Kobo called `pickel' that dumps images in a particular pre-converted raw format to the display. You could possibly use it, in combination with another program to convert the PDFs to a series of raw image frames, to do the job. For this to work you'd have to kill the `nickel' app so it didn't try to refresh the screen. Doing so would disable all the buttons.
I have no developement experience with the original bluetooth-enabled Kobo, so I can't say for sure on that site, but I wouldn't be too surprised if it used the standard bluez stack. If it does, you can probably use the usual bluetooth OBEX tools - obexftp etc - for transfers.
All of this is moderate-level embedded linux work, which means extremely advanced compared to usual Linux stuff. I do not suggest attempting it unless you're willing to spend a LOT of time learning shell scripting, how bluetooth works at a low level on Linux, how the boot process works, serial console access, extracting and loopback mounting disk images, etc etc. You'll also have to open your Kobo, since the original Kobo is different to the one I've been writing about and you'll have to make your own changes to enable usbserial access to give you a console.
I suspect you'd be better off picking up a cheap LCD photo frame or the like.