Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nanoemoji build chain with color font outputs for desktop and web #422

Merged
merged 99 commits into from
Jan 13, 2023

Conversation

JeppeKlitgaard
Copy link
Contributor

@JeppeKlitgaard JeppeKlitgaard commented Dec 8, 2022

Introduction

This PR continues the work of @mavit in #260 and fixes the following issues:

Fix: #93 (though this has other discussions in it as well, such as fixing "black fillings")
Fix: #413
Potentially fixes: #339

TODO

Before merge:

  • Verify fonts work for different platforms and systems (see comment)
  • Fix font/README.md
  • @b-g Verify that you can also run the new tooling. It should be done the way it usually was done. (helpers/generate-fonts.sh)
  • Make SVG+COLR fonts using maximum_color
  • Add this information to README.md
  • Update fonts again
  • Fix bug reported by @b-g

After merge:

  • Make new issue for investigation into ascender/descender sizes
  • Make new issue for investigation into whether zopflipng should be disabled
  • Make new issue tracking COLRv1 support and deprecation of cbdt, sbix, etc
  • Change over font-builder docker image, needs @b-g or permissions. The Dockerfile is found in helpers/docker
  • Close Build COLR TTFs with nanoemoji #260

Content

This PR entirely removes the existing scfbuild tooling for rendering fonts. scfbuild is old and abandoned whereas the replacement, nanoemoji, is actively maintained and implements a lot of features that result in smaller font file sizes and wide compatibility.

We generate fonts for all the formats described below and it is up to the user to choose one that works for their application. A guide will be given in README.md in the future.

Notes

nanoemoji does a lot more work under the hood than the scfbuild tooling did and it takes substancially longer to run. On my machine, the initial rendering of the fonts takes more than an hour. Luckily nanoemoji has a very intelligent build system and renders much quicker for subsequent runs and automagically detects which files have changed and rerenders just those.

Font formats

nanoemoji allows us to render color fonts using a wide variety of formats. A quick overview of them is given here:

  • Black
    • glyf: Standard format that is widely supported
  • Color
    • cbdt: Bitmap format, works well for Android and old Chromium-based systems (< 98)
    • colr_0: Widely supported color format that works in all modern browsers, but lacks some support for advanced features like gradients
    • colr_1: The emerging standard for color fonts, but not yet supported by Safari and many desktop applications
    • sbix: Apple-centric format that is supported in Safari and some MacOS apps
    • svg: Previously preferred and championed by Apple and Mozilla. Not supported by Chromium-browsers, but works in Safari and Firefox

Support

For an overview of which applications support which formats, have a look at colorfonts.wtf, caniuse/colr, and caniuse/colr_v1

Demo

I've now updated my own site to use the generated OpenMoji fonts. Those who are particularly curious can have a look here: https://jeppe.science/

The emojis have a bit more space around them than the twemoji I was using before. This means that the emoji become quite small in regular text and can be somewhat hard to make out, in my opinion. This can possibly be changed by modifying the ascender and descender sizes?

JeppeKlitgaard and others added 30 commits December 6, 2022 23:39
This is done via the container at https://gitlab.com/mavit/nanoemoji-container.

For me, the resulting COLR TTFs work in Firefox and in the Gnome desktop
on Fedora 32 (although they display rather small).

Caveats:
- It may be premature to replace `scfbuild` for SVG builds.  If we decide to go ahead without `scfbuild`, then we’ll want to remove the remaining files related to it.  If we decide to keep `scfbuild`, then we’ll want to revert those parts of this patch.  In either case we’ll want to update the readme to link to the new outputs.
- I’ve tested this with Podman, not Docker.

Relates to hfg-gmuend#93.
Unlike Podman, Docker only supports SELinux relabelling with --volume, not --mount.
Some of them still overspill slightly.  Perhaps scaling by 1.3 is a bit too much?
This yields TTFs of about a quarter of the size of those generated by scfbuild.
This makes quoting more obvious and straightforward, but also ensures that variable substitutions are done inside the container, so we know the Bash version is recent enough to support the `${var^}` feature.
- Itʼs suggested in googlefonts/nanoemoji#157 that we donʼt apply scaling by transforming the root `svg` element, but rather by enclosing the body of the SVG in an element that applies the transform.
- Recent nanoemoji versions are configured with a file rather than on the command line.
Make use of the new `--ascender` and `--descender` options to control the size of the characters.
@JeppeKlitgaard
Copy link
Contributor Author

I am running the builder now, though perhaps @b-g could rerun it on his machine to confirm everything is working once I've pushed the last commit and removed the WIP marker from the PR (likely tonight).

@JeppeKlitgaard JeppeKlitgaard marked this pull request as ready for review January 6, 2023 21:19
@JeppeKlitgaard
Copy link
Contributor Author

@b-g this is ready to merge now. You may want to try and run the builder on your end again, though nothing substantial has changed and it is working nicely for me. The biggest issue is definitely the build times, though with cache it isn't terrible and it only needs to be done when making new releases.

@JeppeKlitgaard JeppeKlitgaard changed the title [WIP] Nanoemoji build chain with color font outputs for desktop and web Nanoemoji build chain with color font outputs for desktop and web Jan 6, 2023
index.html Show resolved Hide resolved
@JeppeKlitgaard
Copy link
Contributor Author

It would be helpful to merge this before too many more changes, as rebasing can be a bit laborious. Should ideally be merged ahead of #431, for example.

@b-g
Copy link
Member

b-g commented Jan 8, 2023

@JeppeKlitgaard Many thanks! Near there! Yay!

Comments:

  • Shellcheck reports issues in helpers/generate-fonts-runner.sh, please fix.
  • I had quite a few merge conflicts, tried to resolve them. Should be good now ... but please double check.
  • Currently running npm run generate-font ... looks good so far. Will report once the build run is finished.

@b-g
Copy link
Member

b-g commented Jan 8, 2023

  • npm run generate-font finished. Yay! (Runtime ca. 3-4h on my mac)

@JeppeKlitgaard
Copy link
Contributor Author

JeppeKlitgaard commented Jan 8, 2023

@b-g I've fixed the shellcheck warnings. I assume running npm run generate-font resulted in new font files – could you pull my changes for the shellcheck warnings and then commit the new fonts and push that? Alternatively I am happy to run the builder again tomorrow afternoon and do the commit the final font files after which we should be ready to merge this PR.

I thought I had rebased off of master and fixed any merge conflicts, but I must not have done that. I've had a look at the merge resolution you did and it all seems fine to me – I manually had a peek at most of the SVGs and the other file changes seem to be correct as well. Apologise for the inconvenience there!

@b-g
Copy link
Member

b-g commented Jan 10, 2023

Hi @JeppeKlitgaard,

npm run generate-font resulted in new font files

Yes, but I was slight confused as the generated files only appeared in the /build/fonts but were not copied automatically to the font folder.

Also there very last line of the entire build process seems to report an issue rm: cannot remove '/mnt/build/fonts/OpenMoji-color-colr0_svg': No such file or directory. Please find attached my entire log: Terminal Saved Output.txt

Hence what about the following approach:
We merge the current state of the PR, even though still a bit WIP ... it seems to be easier to iterate with the changes in in place in the master branch e.g. to put the docker publishing in place. Just it would be good to finish up quickly to not leave master in an in-between state. OK / do you have time in the next days?

@JeppeKlitgaard
Copy link
Contributor Author

JeppeKlitgaard commented Jan 10, 2023

@b-g This is definitely my fault. This bug was introduced with the multi-format font support I made over the weekend and since the build is so a fairly lengthy affair especially without cache I never did a full-rebuild which would have revealed this bug. I will push for a fix for this before the merge. It would probably be good if I could do a full rebuild myself (i.e. deleting build and running npm run generate-font) before the merge as well. If I remember I will do this overnight.

When that is done I would feel fairly confident that the building should just work and I agree that merging this would be a good idea. Docker publishing and multi-arch builds would be a good fit for a separate PR.

@JeppeKlitgaard JeppeKlitgaard marked this pull request as draft January 10, 2023 12:43
@JeppeKlitgaard
Copy link
Contributor Author

JeppeKlitgaard commented Jan 11, 2023

@b-g I have now done a full rebuild from scratch (i.e. deleting ./build and the fonts in ./font and everything seems to be working.

This should be mature enough to merge now. Note that merging this will close 3 issues (see right sidebar). The issues seem to have sprawled a bit in their discussions, so not entirely sure this PR actually fixes them entirely. Might be worth making new issues for the things not fixed by this, or alternatively not closing those issues.

@JeppeKlitgaard JeppeKlitgaard marked this pull request as ready for review January 11, 2023 18:42
@b-g b-g merged commit 6251886 into hfg-gmuend:master Jan 13, 2023
@b-g
Copy link
Member

b-g commented Jan 13, 2023

@JeppeKlitgaard gave it a go as well! Runs now smooth! Many many thanks, also to @mavit! Merged! Yay!! 🎉🎉🎉!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants