[p5.js 2.0 RFC Proposal]: Typography module revamp

[limzykenneth]

Increasing access

Typography module enables text based work to be done in the canvas, creating more possibilities of interaction between free drawing in pixels and text. There are also opportunity for more creative manipulation of text and type once typography module has been stabilized and made up to date.

Which types of changes would be made?

• Breaking change (Add-on libraries or sketches will work differently even if their code stays the same.)
• Systemic change (Many features or contributor workflows will be affected.)
• Overdue change (Modifications will be made that have been desirable for a long time.)
• Unsure (The community can help to determine the type of change.)

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

What's the problem?

Currently the typography module feels a bit brittle with many of the functions around text alignment, width/height measurement, text wrapping, etc, all undergoing fixes at one point or another. These fixes increases the technical debt the module has and also may have hidden edge case bugs (perhaps when using the module with non-English text for example) that we have not found yet.

In addition, for the textToPoints() function and drawing text onto a WebGL canvas, p5.js is currently relying on opentype.js which does not seem to be actively maintained. opentype.js also adds a very significant file size footprint in the overall p5.js bundle.

What's the solution?

There are two main interconnected parts to this:

• Refactor/rewrite/redesign the typography modules to be more internally consistent and possibly changing external API if it made sense.
• Remove, replace, or otherwise improve the use of opentype.js so that features such as textToPoint() and drawing text on WebGL canvas still continue to work with or without opentype.js. The priority will be to have either an up to date dependency or if at all possible not have any dependencies, reducing file size and future proofing the dependency list.

One possible way to start is to leverage browser text rendering API as much as possible, much of CSS and possibly the browser's own text rendering engine can potentially be leveraged to do much of the work (eg. loading fonts into CSS works as a way to load fonts in canvases). Some feature, especially those we rely heavily on opentype.js as mentioned above will likely not work out of the box with browsers so will need new solutions there.

The typography module should also be test to work across as many language scripts as possible without extra effort from the user.

Pros (updated based on community comments)

• Reduce bundled library size
• Reduce reliance on unmaintained projects
• Easier to maintain internal code and easier to use external API

Cons (updated based on community comments)

• Some features can prove to be very difficult to do

Proposal status

Under review

[davepagurek]

For some more context, we looked into some alternatives here: #6391 (comment)

How much do we care about feature compatibility between renderers? Switching to using native canvas APIs is very clearly a win for 2D mode. I'm not sure there's a way to get this to work in WebGL very easily though, and it will probably lag behind for features like variable fonts. OpenType.js doesn't even support that, but it still seems like unfortunately maybe the best option out there. So one path forward could be to let 2D and WebGL diverge in their feature sets, and only include OpenType.js in builds that include WebGL.

[limzykenneth]

I just found gl-text, perhaps we can have a look at their approach?

[davepagurek]

gl-text uses font-atlas, which is kind of like if we rendered all the characters of a font to a 2D canvas and then used those as textures in WebGL. The main reason why we haven't been doing that in p5.js is that the resolution of texture that you need depends on how close the text is to the camera, which may vary from frame to frame as things move, possibly resulting in pixelated text. That tradeoff is potentially OK for some, but right now we've opted for a vector drawing approach which will be crisp regardless of the camera. The downside is that it needs the Bezier curve information to do that, which needs a font parser.

font-atlas-sdf is a step closer, because you can render an SDF at higher resolutions and it will still be crisp, but it still will result in a lack of accuracy once you've zoomed in farther than the original resolution. Personally I'm more OK with this tradeoff compared to a regular font texture atlas, as it also would let us add stroke to text in WebGL mode, but this is still a bit of a tradeoff in accuracy compared to the current implementation.

Both could potentially work with variable fonts, although probably more slowly than in 2D mode when you adjust font variables since it would have to recreate the atlas each time something changes.

[limzykenneth]

Yeah that made sense. I had a look into how Three.js does it and it seems ultimately they are also relying on opentype.js to preprocess font files into points then geometries first.

An alternative that may be easier or much harder is to fork opentype.js to bring it more up to date but we may not have enough expertise or availability amongst ourselves to get new features into it in the near term. The best I can think of for forking is to make it use latest JS features so that we can potentially tree shake the library during bundling to make it smaller, for variable fonts I may need to read up on the font file spec.

[davepagurek]

Actually, looks like in the past few weeks they added support for variable fonts! opentypejs/opentype.js#699

[limzykenneth]

We're gonna need a new release on NPM for it.

[davepagurek]

haha hopefully by the time we get around to implementing it they'll have released a new version! Worst case, you can technically use a git dependency via a URL + commit hash in package.json, but my experience doing so has been kinda buggy when you update the hash (removing and re-adding the dependency seemed the most consistent way to get it to update), so it wouldn't be a long-term solution.

[nickmcintyre]

Could TinySDF be helpful?

If we were to use this library or a similar 2D canvas-based approach, then a (maybe too) simple fix for zooming could be to render the text much larger by default. Size/zoom could be adjusted by scaling the 2D canvas. There'd be a tradeoff with the additional memory used for the larger 2D canvas and the math used for scaling.

[davepagurek]

Could TinySDF be helpful?

definitely! Any SDF texture will be more useful to us than a regular texture because we can draw that at different sizes and still get a crisp line, and increasing the size of the texture will just give us a more accurate line at large zooms, which seems like something we could reasonably make a function for to let people set it higher if they need that. If we're looking to ditch OpenType then that's probably the best compromise approach for WebGL.

It probably will be slower at doing variable font changes than a 2D canvas though since it has the additional step of running the rendered character through a distance transform before it can be used for rendering, but I think that's also a pretty OK tradeoff. People can always render text to a 2D p5.Graphics and draw it into a 3D main canvas if they want better variable font speed at the cost of having to manually handle scaling/pixel density.

[agrshch]

This is amazing that there is a plan to remake the typography module, it's much needed! Thank you so much for bringing this up!
As a designer, I would be incredibly happy to have within p5js direct access to opentype features, variable font axes, letter spacing, built-in kerning, etc. This will be a really huge leap forward!

[davepagurek]

The kerning/spacing thing is also something important we haven't discussed in depth here yet. For 2D mode we can rely on native tools, which is great. For other renderers, currently we'd need to be doing all of that manually, finding a library to do that (last I looked into it there weren't too many js options, although cross-compiling a native library is an option), or having a reduced feature set in non-2D renderers. Anyone have any thoughts on the time/consistency/filesize tradeoffs of those options?