Discussing the way to write a multi language menu

[davetcc]

Often referred to as i18n, tcMenu 3.2 will support the ability to generate multi-language menu applications. In the first instance, it will be done through the use of a TC_LOCALE_?? flag, that for example to enable French strings would require compile option TC_LOCALE_FR to be set. This also localizes nearly all strings inside core tcMenu as well.

This purely focuses on the flow to add locale Strings to a menu project. First we start tcMenu and enable localization. To do so from the "Code" menu we pick "Configure Locales":

From this dialog we can enable the locales that we want, and the designer will write out the source resource bundles for us.

Although each language can be edited within the designer, it is actually quite easy to edit the source bundle files from any editor, they are simple properties files.

Above is the example layout of an i18n project. Notice the i18n directory with the source bundles in it, an example bundle is loaded. The "default" bundle has no extension and it is better to keep this as English language, to stay in keeping with the core.

Notice the use of a generated folder where all the generated source can now be optionally kept. The theme (which becomes part of user code, and also the same for BSP settings on STM32 LTDC) are within your code, as they don't get overwritten again.

[davetcc]

It should also be said that when this support is off, the designer will act exactly as it did before.

[vzahradnik]

It looks really well. I have two points though:

• I suggest to implement language support in the format like this: en_us (language_region). This is a standardized way used everywhere and there's a reason. For example, there isn't just one German language but you have variants, like Swiss (de_CH), Austrian (de_AT), etc. Similarly to English. Even Windows has British (en_GB) and US (en_US) locales. There is a need, especially for these global languages, to customize the strings for the specific region. If users don't select a region, then the same language is used for everyone.
• Another suggestion is something we can do over time as it's not that easy as extracting strings. In your current example you show that users can translate unit names. This is good. But let me give you an example with the temperature. In Europe we mostly use degrees Celsius where in the US they use Fahrenheit. Similarly, metric (meters) vs imperial (yards) units. Probably this is something a project owner will need to deal with in the code but perhaps TcMenu could provide kind of helper methods to help with the process. Conversion between Celsius and Fahrenheit will be a very common use case.

Thinking about the second usecase, i have some idea. Basically the defined menu item is not aware of the units used. So in theory we can just take the value as a base number and hook up an optional conversion method which will trigger based on some conditions, like language_code flag. Internally we'll still work with the base units, just the UI representation will need to do coversions as needed (change via UI or change via the code).

[davetcc]

Thanks for the feedback.

For point 1, agreed, the Java Locale class supports that the support is based on is very complete (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Locale.html) and there should be nothing that prevents someone creating even the most complicated of Locale with even language, country and variant, the only problem is how we would express that in the compile flag.

I would think just: TC_LOCALE_FR_CA for example, do you think that would work? Let's even say there was one with a variant, such as TC_LOCALE_LL_CC_VV would probably work. I'll look at adding at least the country support into the UI before 3.2 BETA.

For point 2, I think that anyone who solves such problems could create another helper in the "extras" folder that would then be available for all.

[vzahradnik]

To point 1, yes that would work. And to point 2, sure, one can create such helpers. See my updated comment just as one possible suggestion.

[davetcc]

Locales of Language + Country are working now on master. I just have a little more work to do in the code generator and a few final pieces of UI work to do, and then I'll create a beta once it is working for the main test cases. All possible cases should work once finished including all C++ generators and the Embedded Java Raspberry PI generator.

This is going to need a lot of testing, more than I will be able to do. I think what's best is that I create a beta release, and as many people as possible try it out and make sure it is working.

[vzahradnik]

Sounds good.

[davetcc]

So for translation I think the things that are needed are:

• Core strings in TcMenu - DONE and in 3.1
• The name of all items
• The unit for analog items
• The value names for enum items

I don't think there is anything else dates and times are already quite configurable, and we can easily make them more so if needed.

[davetcc]

So also the application name can also be internationalized, simply in the UI add i18n key into the resource bundle and put a "%" at the beginning, then add the entries into the resource bundle yourself. You can also add any other entries in there for your app, they follow a simple convention to convert from properties into code.

See the example that was generated by the in-development version of the designer -

https://github.com/davetcc/tcMenuLib/tree/master/examples/arduino32/picoAdafruitDashboard

[davetcc]

The menu item names, units, and enum names are done automatically by designer with a simple locale switch available next to the item name. This will need a lot of testing.

[vzahradnik]

Looks good!

[davetcc]

I've pushed out TcMenu 3.2 BETA today, I've done a fair bit of testing on it myself already, but the changes are pretty substantial, and will need more testing than a usual release.

[davetcc]

Back up your project (or put a tag in the git timeline) so that you can get back to what you had before trying.

[vzahradnik]

How can I get to the beta? By the way, I just tagged my repo but it was just a coincidence :)

EDIT: I found it.
https://github.com/davetcc/tcMenu/releases

[vzahradnik]

First problem:

Compiling .pio\build\hentower\lib9b9\Adafruit Unified Sensor\Adafruit_Sensor.cpp.o
src/hentower_menu.cpp:27:106: error: 'TC_I18N_MENU_78_UNIT' was not declared in this scope
 const PROGMEM AnalogMenuInfo minfoOutsideHumidity = { "Outside hum", 78, 0xffff, 100, NO_CALLBACK, 0, 1, TC_I18N_MENU_78_UNIT };
                                                                                                          ^~~~~~~~~~~~~~~~~~~~
src/hentower_menu.cpp:31:104: error: 'TC_I18N_MENU_77_UNIT' was not declared in this scope
 const PROGMEM AnalogMenuInfo minfoInsideHumidity = { "Inside hum", 77, 0xffff, 100, NO_CALLBACK, 0, 1, TC_I18N_MENU_77_UNIT };
                                                                                                        ^~~~~~~~~~~~~~~~~~~~
src/hentower_menu.cpp:107:97: error: 'TC_I18N_MENU_44_UNIT' was not declared in this scope
 const PROGMEM AnalogMenuInfo minfoContrast = { "Contrast", 44, 144, 100, ContrastChanged, 0, 1, TC_I18N_MENU_44_UNIT };
                                                                                                 ^~~~~~~~~~~~~~~~~~~~
src/hentower_menu.cpp:131:100: error: 'TC_I18N_MENU_33_UNIT' was not declared in this scope
 const PROGMEM AnalogMenuInfo minfoTargetHumidity = { "Target hum", 33, 57, 100, NO_CALLBACK, 0, 1, TC_I18N_MENU_33_UNIT };

I use all your libraries from master. This seems to be a generator issue. Do I need to configure something extra to make this work?

[vzahradnik]

So this issue was resolved after the default lang file was properly generated. See issue below. I was able to compile my project now.

[vzahradnik]

Second problem:
In TcMenu Designer, after clicking on a item, it is not shown on the screen to edit. The code is attached below.

    {
      "parentId": 6,
      "type": "subMenu",
      "item": {
        "secured": false,
        "name": "%menu.71.name",
        "variableName": "AnalogInputs",
        "id": 71,
        "eepromAddress": -1,
        "readOnly": false,
        "localOnly": false,
        "visible": true,
        "staticDataInRAM": false
      }
    },
    {
      "parentId": 71,
      "type": "analogItem",
      "defaultValue": "0",
      "item": {
        "maxValue": 65535,
        "offset": 0,
        "divisor": 1,
        "unitName": "",
        "step": 1,
        "name": "Photoresistor",
        "variableName": "PhotoresistorValue",
        "id": 72,
        "eepromAddress": -1,
        "readOnly": true,
        "localOnly": false,
        "visible": true,
        "staticDataInRAM": false
      }
    },

[vzahradnik]

Default project-lang.properties contains all strings but the project-lang_en_US.properties is missing Photoresistor entry due to this bug. I resolved this by copying the default property file as en_US.

[vzahradnik]

Default lang file is generated empty (with a comment). In order to populate this file, I had to modify arbitrary entry and save the project file. Afterwards all entries were exported.

[vzahradnik]

Last but not least, language files inside TcMenu library. Currently there is support for 4 languages but users don't know about this option.

They might be lucky. As long as they specify just the language code, not the country code, a language is activated automatically. In the language_select.h are the following defines:

#elif defined(TC_LOCALE_FR)
#elif defined(TC_LOCALE_SK)
#elif defined(TC_LOCALE_CS)
#elif defined(TC_LOCALE_EN)

What I have instead is language code like

#if defined(TC_LOCALE_CS_CZ)
#if defined(TC_LOCALE_SK_SK)

The obvious solution is to include both flags but it's too verbose. It would be great if we could detect and apply proper language even though the country is set. Like if there is not country specific language, we'll fallback to language specific file TC_LOCALE_CS and if not found, fallback to default TC_LOCALE_EN.

I know the library uses pre-processor directives and I have no idea whether such a detection is possible. If not, it should be at least mentioned in the documentation.

[davetcc]

I would look at the code, it's pretty straightforward and you'll soon see how the precedence works until I have time to document, it uses the same rules as Java precedence. Default lowest (but must exist), then language, then language-country. So the lookup rules in the properties are:

For each property in the default properties (IE everything must have a default entry):
    Does the language, country mapping exist for that property -> use that
    Otherwise does the language mapping exist for that property -> use that
    Otherwise use the default property value
End

Unfortunately, at the moment I don't have as much time as I'd like to work on it as I've burnt up what I had trying to get it this far. I would suggest you take a look at the code for now if you want to better understand it.

https://github.com/davetcc/tcMenu/blob/master/tcMenuGenerator/src/main/java/com/thecoderscorner/menu/editorui/generator/core/CoreCodeGenerator.java#L211

and

https://github.com/davetcc/tcMenu/blob/master/tcMenuJavaApi/src/main/java/com/thecoderscorner/menu/persist/PropertiesLocaleEnabledHandler.java#L36

[davetcc]

BTW EN is usually the default if you want it to line up with how the locale stuff works in the library for the core strings.

[vzahradnik]

I know EN is usually the default but from what I observed elsewhere, even though the EN is the default and used as a template/fallback option, projects still generate English lang files. So that is what I did. At the moment these two files are identical.

[davetcc]

If there's any doubt, look at the example I converted - https://github.com/davetcc/tcMenuLib/tree/master/examples/arduino32/picoAdafruitDashboard

[vzahradnik]

How can I add extra fields in the property file? For example a text for dialog. I tried adding this manually into the file but the generated code doesn't include this line.

[davetcc]

Just close the project in designer and open the properties using a regular text editor. The project should be closed in designer before you do that though, that is the only restriction at the moment.

[vzahradnik]

OK, I'll try and report back.

[davetcc]

The only other thing to bear in mind, is that the default properties file is the driver for generating all generated output, whatever property you add must always have an entry in that file. The designer UI always makes sure that is the case.

[davetcc]

Let me try and create another project from scratch with Locale support enabled at some point during the week, in case there is something specific to enabling on a new project.

[davetcc]

To be honest, this has really taken a lot longer than I expected, and left many other projects stalled for quite some time. We just need to get this over the line in working order, with as close to what's there now as possible. I can see the value for others, but realistically it has limited value for projects I have at hand right now so is just a direct cost on my time.

[davetcc]

Yes use IntelliJ IDEA (community should be fine), best to use JDK 20, as that's the latest, designer always builds with the latest.

For now ignore the API and embedCONTROLCore, they will get pulled in for you automatically from maven central, and unless you need to fix an API level issue, there are a couple of complexities with the API, mainly that the maven build for it needs a GPG key.

To pull in the projects first create an empty project, then go to Project Structure, and click the "+" toward the top left, from the menu select import module, then choose the pom.xml from each of the two projects. That should import both projects in a workable state.

Before you can run it, for the first time you need to run the maven install build once on designer with the -DskipTests option.

To run set up a run configuration with the following settings:

Extra VM options: -Ddevlog
Working directory: usually I choose the tcMenu top level directory

That should be it, it should run and work with that.

[davetcc]

If you wanted to edit Plugins -

If you wanted to run up with the ability to edit the plugins easily, then you can do the following. Plugins are expanded from the app Jar file into ~/.tcmenu/plugins, the plugins are nothing more than a copy of what's in the repo tcMenu/xmlPlugins directory. So you can simply symlink the three plugin directories into the .tcmenu/plugins directory.

If you do this be sure to put a file called .development into the .tcmenu/plugins directory as it stops the designer from trying to update them.

[davetcc]

The best JDK to use is probably Liberica OpenJDK 20 as if you use the Full version it has JavaFX built in.

[davetcc]

In fact, I'll write up how to load the project into the IDE, as it would be a useful guide for others.

[vzahradnik]

I'm sure it will be useful. Perhaps some of them will be more likely to contribute if this information is available.

[davetcc]

Should you need to include the API. As the API and embedControlCore are in Maven central, the Maven build needs a GPG key, given you'll only be building it for local use, any GPG key will do, there are loads of discussions about this online.

[vzahradnik]

Update:

So I spent the Weekend setting up the dev environment with JavaFX. I got to the point that I can build the packages but IDEA still has some issues with running the app. I will solve this. Just to let you know that I plan to fix these bugs I found.

If you decide to write that post about preparing the dev environment, perhaps I could provide a feedback as I deal with it right now. One thing I noticed already is that if I include external JavaFX libraries, I have to uncomment the dependency in the pom files. You have a note there.

I will post here once I have something new.

[davetcc]

Feel free to ping me by email with how far you've got and I'll try and help you move forward.

[vzahradnik]

Heureka! I have TcMenu build from the sources running in IDEA.

There were a couple of issues.

• First I added JavaFx to the pom files
• Then I had to set Java language features to 20. Especially the method Locale.of() is supported since API 19 and above.
• Finally, I had to tell the javac compiler to compile imported TcMenu modules with Java 20 bytecode. It was pre-set to 17 for some reason.

I hope these notes help documenting that to others. Now I can finally have a look at the i18n bugs,

[davetcc]

Great news. I’m going to document this process at the weekend. Once I’ve done that maybe you can go through and add any missing points.

[vzahradnik]

That's what I though too. Just send the link and I can go through it.

[davetcc]

Btw if you run the designer tests locally there are some UI automation tests in the test source ( within the ui package) that are quite difficult to run. I’m going to be breaking these out into another project same time as doing the documentation. I just usually use them once per release to smoke test the UI.