Situation

Currently, loading our SDK takes approximately ~350ms (2021 M1 Max), incurred primarily by loading numpy, PIL, and PyArrow.

python -X importtime -c "import rerun" 2> import.log
tuna import.log

This is particular impactful when running python -m rerun to launch a viewer (since #4053, using the rerun script is much faster), but probably also impacts (simple) logging script.

Currently, ~50% of the loading time is incurred by Numpy+PyArrow. This is probably unavoidable as no use-case realistically exists that doesn't require any of those, at least indirectly (it's pervasive in our implementation).

However, our using of a large number of files does incur a cost that is non negligible:

In the case of datatypes, about 5-6% are fixed cost (numpy stuff, base class, etc.), but the remaining 10-11% is the sum of all datatypes, lots of which might not be actually need by user code. The same story applies for archetypes (~4%) and components (~4%). Today, this is not much (<20%, ~60ms?), but with a grow in time as our API becomes more rich.

Possible solutions

Multiple project make use of __init__.py-level __getattr__() to implement lazy loading:

• https://github.com/pallets/werkzeug/blob/main/src/werkzeug/__init__.py
• https://github.com/pydantic/pydantic/blob/main/pydantic/__init__.py

Something like this, combined with codegen, should be feasible without too much pain.