Jarnal - Handwriting Recognition

by David K. Levine

Handwriting Recognition

The pen, but not the highlighters support handwriting recognition. ONLY PRINTED CHARACTERS ARE SUPPORTED, not script, cursive, or any sort of handwriting where the letters are all connected together. You can either recognize all handwriting on a page Tools | Recognize Page or you can turn on the recognition engine Tools | Recognize. Recognized characters are typed in the most recently selected text box. Jarnal comes with a small dictionary of lower case Latin letters; you can build your own dictionaries to add to this, or replace it. User dictionaries are called jarnal.recog and must be located in your home directory. If you aren't sure where this is, take a look at the Help | About box in Jarnal.

Other Options: On the Tools | Recognize Menu you can clear the recognition dictionary (to start over), you can Undo Recognition (also available from the Context Menu | Type | Undo Recognition) or you can enter training mode to build your recognition dictionary. Undo recognition works a little better than simply hitting the undo button when in recognition mode: it undoes the last letter recognized and all the strokes that went with it, plus it reinitializes the recognition engine properly.

Building a Recognition Dictionary: When in training mode, any stroke that is not sufficiently recognized causes a dialog box to pop up. You type what it is you are trying to write in this box. Note that your are not limited to single characters; if you are ambitious you can build an entire system of shorthand, where single small strokes write entire large sentences.

How it Works (or Doesn't): Each stroke is normalized to a common scale (so a very large B looks the same as a very small B). Then a whole bunch of different methods are used to try to determine which stroke/strokes in the dictionary match whatever you are drawing, and they are averaged together to get a score. Then a table of probabilities of pairs of letters is applied to guess, for example, that if the previous letter was a q, the next letter probably isn't going to be an x. (The mini-Jarnal does not do this guessing or spacing - see below.) Spacing is handled by seeing how far apart you drew your letters, and also using the table of letter pairs - it turns out, for example that in formal writing at least, few English words end in the letter i, so if you try to write hi, you may find the results surprising. The table of pairs is contained in images/pairs.txt in the Jarnal.jar file - and it can be replaced by another table. Here is a PHP script that reads a text file in the same directory called "sample.txt" and writes (to the screen) the corresponding table of conditional probabilities. So use php -q analyze.php >pairs.txt, to get your pairs.txt file. Feel free to replace the existing pairs.txt with your new one. The default table was generated by downloading from Gutenberg a subset of their most popular books meeting certain criteria, such as having been originally written in English and being available in ascii format, sticking them together in a large file called sample.txt and running analyze.php. The books actually used were Ulysses, Relativity: the Special and General Theory, The Adventures of Sherlock Holmes, and The War of The Worlds, which should be random enough for practically anyone.

Source Code: The source is Janalyze.java. This has no dependencies on any other part of Jarnal, so can be dropped into any other project by simply changing the first line of the file to reflect the correct package. However, I haven't documented the API yet, so you'll have to guess this by reading the code.

The mini-Jarnal: if you open Jarnal from the command line with the switch -mini a mini-Jarnal will open. The mini-Jarnal is highly customized for handwriting recognition. It supports enough different sets of symbols that you can actually write something and have it recognized. Recognized text is copied to the clipboard so you can transfer it to other programs. There is also a calculator mode that treats whatever was recognized as an arithmetic expression and attempts to evaluate it, copying the result to the clipboard. The philosophy of the mini-Jarnal is somewhat different than that of text recognition in Jarnal proper, although both share the same basic engine. In Jarnal it tries to make an educated guess at what you meant to write, and if you print clearly, it will get it right to a surprising extent. (And if you are sloppy it will produced total gibberish.) The mini-Jarnal is designed so you can write exactly what you want, and doesn't use nearly as much guesswork.