language-features.md

Contributing to a Language

Improving Yomitan's features for the language(s) you are interested in is pretty simple, and a great way to help yourself and others. This guide will help you get started.

Adding a Language

If your language is not already available in the Language dropdown, here is how you can add it with just a few lines. As an example, we'll use PR #913, where a first-time contributor added Dutch.

// language-descriptors.js
{
    iso: 'nl',
    name: 'Dutch',
    exampleText: 'lezen',
    textPreprocessors: capitalizationPreprocessors
}

1. Look up the ISO 639-1 code for your language. If it is a rarer language it might not have a ISO-639-1 code - if that's the case, use ISO 639-3.
2. Place your language in the languageDescriptors array in language-descriptors.js. The languages are sorted alphabetically by ISO code.
3. The example text is usually some form of the verb "to read" in your language, but it can be any example you feel is good. This will be shown in the preview popup on the settings page.
4. If your language uses the Latin or Cyrillic script, or another script with capitalization, you will want to use the capitalizationPreprocessors. We'll cover this part in more detail a bit later. The textPreprocessors field can also be left out.

When in doubt, look at the other languages in the file for ideas. The same applies to language-descriptors.d.ts:

// language-descriptors.d.ts
nl: {
  pre: CapitalizationPreprocessors;
}

This is just for some type safety. The first key is the ISO code. Most languages will then only have a pre key (the other one is post), and its value is the type of text preprocessors you used in language-descriptors.js. Use the TypeScript operator & as needed. If you didn't use any text preprocessors, you can set the value to Record<string, never>.

That's it! Your language should now be selectable from the dropdown, and may work perfectly fine already. If you don't already have a dictionary to test with, check out Dictionaries. For more advanced features, read on.

Language Features

You should first have the repo set up locally according to the instructions in the contributing guidelines.

A language descriptor in language-descriptors.js has several optional fields for more advanced features. We've already mentioned textPreprocessors, but there are also languageTransforms, textPostprocessors, and isTextLookupWorthy. Let's go through them (see also the full type definition in language-descriptors.d.ts).

Text Preprocessors

The scanned text may not exactly match the word in the dictionary. For example, an English dictionary will likely contain the word "read", but the text may contain "Read" or "READ". To handle cases like this, we use text preprocessors.

// from language.d.ts
export type TextProcessor<T = unknown> = {
  name: string;
  description: string;
  options: TextProcessorOptions<T>;
  process: TextProcessorFunction<T>;
};

Every text preprocessor has:

• A name and description
• An array of options, most commonly just [false, true], that control the behavior of the process function.
• A process function that takes a string and a setting and returns a string

Here are the CapitalizationPreprocessors used in the Dutch example:

/** @type {import('language').TextProcessor<boolean>} */
export const decapitalize = {
  name: "Decapitalize text",
  description: "CAPITALIZED TEXT → capitalized text",
  options: basicTextProcessorOptions, // [false, true]
  process: (str, setting) => (setting ? str.toLowerCase() : str),
};

/** @type {import('language').TextProcessor<boolean>} */
export const capitalizeFirstLetter = {
  name: "Capitalize first letter",
  description: "lowercase text → Lowercase text",
  options: basicTextProcessorOptions, // [false, true]
  process: (str, setting) =>
    setting ? str.charAt(0).toUpperCase() + str.slice(1) : str,
};

When applying preprocessors, each combination will be separately applied and looked up. Since each of these two preprocessors has two options, there are 2*2=4 possible combinations. For the input string reaD, the following strings will be looked up:

• reaD (both preprocessors off)
• ReaD (only capitalizeFirstLetter on)
• read (only decapitalize on)
• Read (both preprocessors on)

Note that the order of text processors can matter. Had we put capitalizeFirstLetter before decapitalize, the 4th string would be read instead of Read.

Letter Variants

A letter or a group of letters may have multiple variants in a language. For example, in German, "ß" can be written as "ss" and vice versa, or in Japanese every kana has a hiragana and a katakana variant. To handle this, we use a bidirectional conversion preprocessor.

// from german-text-preprocessors.js
/** @type {import('language').BidirectionalConversionPreprocessor} */
export const eszettPreprocessor = {
  name: 'Convert "ß" to "ss"',
  description: "ß → ss, ẞ → SS and vice versa",
  options: ["off", "direct", "inverse"],
  process: (str, setting) => {
    switch (setting) {
      case "off":
        return str;
      case "direct":
        return str.replace(/ẞ/g, "SS").replace(/ß/g, "ss");
      case "inverse":
        return str.replace(/SS/g, "ẞ").replace(/ss/g, "ß");
    }
  },
};

These have three options: off, direct, and inverse, and the process function must handle each of them.

Removing Diacritics

In some cases (e.g. German umlauts), diacritics are near-ubiquitous. However some languages (such as Latin, Arabic etc) do not commonly use diacritics, but only in specific kinds of texts (e.g dictionaries, texts for children or learners). In these cases, the dictionaries that Yomitan uses will likely not contain diacritics, but the text may contain them. To handle this, we use a diacritics removal preprocessor.

This kind of text processing is to a degree interdependent with the dictionaries available for the language.

Deinflection Rules (a.k.a. Language Transforms)

Deinflection is the process of converting a word to its base or dictionary form. For example, "running" would be deinflected to "run". This is useful for finding the word in the dictionary, as well as helping the user understand the grammar (morphology) of the language.

These grammatical rules are located in files such as english-transforms.js.

Not all the grammatical rules of a language can or need to be implemented in the transforms file. Even a little bit goes a long way, and you can always add more rules later. For every couple rules you add, write some tests in the respective file (see english-transforms.test.js), and run npm run test:unit. This will help you verify that your rules are correct, and make sure nothing is accidentally broken along the way.

Transforms files should export a LanguageTransformDescriptor, which is then imported in language-descriptors.js.

// from language-transformer.d.ts
export type LanguageTransformDescriptor = {
    language: string;
    conditions: ConditionMapObject;
    transforms: Transform[];
};

• language is the ISO code of the language
• conditions are an array of parts of speech, and grammatical forms that are used to check which deinflections make sense. They are referenced by the deinflection rules.
• transforms are the actual deinflection rules

Let's try and write a bit of deinflection for English, from scratch.

// english-transforms.js
import { suffixInflection } from "../language-transforms.js";

export const englishTransforms = {
  language: "en",
  conditions: {},
  transforms: [
    {
      name: "plural",
      description: "Plural form of a noun",
      rules: [suffixInflection("s", "", [], [])],
    },
  ],
};

This is a simple example for English, where the only deinflection rule is to remove the "s" from the end of a noun to get the singular form. The suffixInflection function is a helper that creates a deinflection rule for a suffix. It takes the suffix to remove, what to replace it with, and two more parameters for conditions, which we will look at next. The suffixInflection is the most common type of deinflection rule across languages.

For the input string "cats", the following strings will be looked up:

• cats (no deinflection)
• cat (deinflected by the plural rule)

If the dictionary contains an entry for cat, it will successfully match the 2nd looked up string, (as shown in the image). Note the 🧩 symbol and the plural rule.

However, this rule will also match the word "reads", and show the verb "read" from the dictionary, marked as being plural. This makes no sense, and we can use conditions to prevent it. Let's add a condition and use it in the rule.

conditions: {
  n: {
    name: 'Noun',
    isDictionaryForm: true,
  },
},
transforms: [
  {
    name: "plural",
    description: "Plural form of a noun",
    rules: [
      suffixInflection("s", "", [], ["n"])
    ],
  },
],

Now, only dictionary entries marked with the same "n" condition will be eligible for matching the plural rule. The verb "read" should be marked as "v" in the dictionary, and will no longer be matched by the plural rule. The entries in the dictionary need to be marked with the exact same conditions defined in the conditions object. The isDictionaryForm field can be set to false, to allow some conditions to be sued only in between rules, and not in the dictionary. In most cases however, it will be set to true.

Now consider the word dogs', as in the the dogs' bones. This is the possessive of a plural noun. We can add a rule for the possessive:

{
  name: "possessive",
  description: "Possessive form of a noun",
  rules: [
    suffixInflection("'", "", [], ["n"])
  ],
},

However, the only conditionOut of this rule, n, does not match any conditionIn of the plural rule, because the plural rules conditionsIn are an empty array. To fix this, we can add a condition to the plural rule:

{
  name: "plural",
  description: "Plural form of a noun",
  rules: [
    suffixInflection("s", "", ["n"], ["n"])
  ],
},

Now the rules will chain together, as shown in the image. Chaining can be very useful (for agglutinative languages it is indispensable), but may cause unexpected behavior. For example, boss will now display results for the word bo (e.g. the staff) with the plural rule applied twice, i.e. it can chain with itself because the conditionsIn and conditionsOut are the same. This leads us to the actual implementation of the plural rule in english-transforms.js:

conditions: {
  n: {
    name: "Noun",
    isDictionaryForm: true,
    subConditions: ["np", "ns"],
  },
  np: {
    name: "Noun plural",
    isDictionaryForm: true,
  },
  ns: {
    name: "Noun singular",
    isDictionaryForm: true,
  },
},
transforms: [
  {
    name: "plural",
    description: "Plural form of a noun",
    rules: [
      suffixInflection("s", "", ["np"], ["ns"])
    ],
  },
],

Since ns and np are subconditions of n they will both match with n, but not with each other. This covers all of the requirements we have considered.

The suffixInflection is one of a few helper functions - you can write more complex rules, using regex and a function for deinflecting. There are examples of this across the language transforms files.

Text Postprocessors

In special cases, text may need to be modified after deinflection. These work exactly like text preprocessors, but are applied after deinflection. Currently, this is only used for Korean, where the Hangul text is disassembled into jamo during preprocessing, and so must be reassembled after deinflection.

Text Lookup Worthiness

Some features include checking whether a string is possibly a word in the language. For example, trying to look up, in an English dictionary, a word written with non-Latin characters (e.g. "日本語") will never yield any results. To prevent unnecessary lookups, an isTextLookupWorthy function can be provided, otherwise all text will be looked up.

Stuck?

If you have any questions, please feel free to open a Discussion on Github, or find us on TheMoeWay Discord at #yomitan-development.