Scripts or algorithms for automatic building of dictionaries based on plain text or wordlist input

[Tulitomaatti]

In addition to English, I also type in other languages, and generally jargon-heavy text. It would be nice to have a tool to generate a chord/shorthand dictionary based on a plaintext file of sample text, or from just a word list with word frequencies included. This would allow using ZipChord in other languages quickly, and should generalize quite well to any specific type of text.

I don't have any background in theory regarding optimal shortcuts or chords, but the following might be something to consider:

• Shorter words get relatively less benefit from chords or shorthands
• Too rare words don't get used as much and thus get less benefit
• Chords should be ergonomically feasible to press - "asdf" is very easy to chord but "zwqx" or "rtbv" or even "tf" are not. (Even better if this could be analyzed given a keyboard layout and/or information whether the user has a staggered or an othorlinear keyboard.)
• Chords/shorthands should be relatively easy to memorize - "weth" for whether makes sense, but "asdf" for "ergonomical" does not.

I tried searching around for such a script or algorithm, but couldn't find anything satisfactory; please enlighten me if this has already been trivially solved somewhere.

[psoukie]

This is a bit outside of the scope of the tool itself. For shorthands, this is obviously easier (I'm including the general approach for how I built the English one in the file, and reprinted below for reference.)

What you are suggesting makes a lot of sense. (Shorter words, rare words ... Completely agree, that's what goes into the considerations.)

For the ergonomically feasible -- I sometimes use a 'mirroring' technique. See discussed under #8. This might, or might not make sense for other people.

You start running into conflicts pretty quickly, and how to best resolve them (again, with all the considerations), is pretty tricky to automate (or to automate to make sense).
 
Also agree on the memorization. I feel that because of the Zipf's law, you don't need a huge number of chords, and because of all the considerations, I created the chord dictionaries by hand. But would love to see some algorithms that would try and do this automatically.

How the English shorthand (not chord) dictionary was built:

This dictionary is based on the list of 5,000 most frequent English words at https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/A_Frequency_Dictionary_of_Contemporary_American_English) which has 4374 entries after removing duplicates.

Sequences of typed keys that trigger the shorthand recognition are separated from the expanded full words by Tab. ZipChord ignores lines without Tab, lines starting with Tab (i.e. where the first column is empty), and text in additional columns.

You can change existing shorthands and add your own.

To make this shorthand dictionary predictable, these default shortcuts adheres to consistent principles:

1. If the word is three characters or less, no shorthand is predefined
2. Doubled consonants (two consonants immediately following each other) are replaced with only one consonant
3. If the word starts with a vowel, the vowel is retained in the shorthand
4. All remaining vowels are removed
5. If the resulting shorthand is already in use or is a word in its own right, no shorthand is predefined