[join-] load all sheets before joins

[midichef]

When joins are done on sheets that are loading, rows can be lost in the new sheet.

repro: seq 10000000 > 10m.tsv; vd 10m.tsv sample_data/test.fixed
While the larger sheet is loading, go to the Sheets sheet with S, quickly select both sheets, & to join, then append. The final sheet should have 10,000,002 rows, but if the join is started quickly enough, it won't.

The requirements for rows to be lost here are:

1. a sheet is still loading when the join is done
2. the slow-loading sheet is on screen
3. the off-screen sheets finish loading before the on-screen sheet does

The problem is how we use BaseSheet.ensureLoaded(). We currently use its return value to decide whether a sheet has finished loading. But we can only do that if it hasn't been called for that sheet before. If it has been called before, it will return None even when the sheet is still loading.

For the sheet on screen, ensureLoaded() is called before, by drawSheet() in mainloop.py. So we cannot use its return value in join.py to guide the decision to call sync().

This PR changes the sync() to be done unconditionally.
It also fixes errors calculating progress while the number of final rows is not yet known (dd1e3fe).
And adds a loading Progress indicator during all the syncs.

[midichef]

Is there any way to test whether a given sheet is fully loaded? Right now I only know of three tests that relate to loading state:

1. sheet.rows is UNLOADED, when loading has not started yet
2. the return value of sheet.ensureLoaded()
3. inside with visidata.ScopedSetattr(self, 'loading', True)

Is it part of visidata's design to have such a state as "fully loaded" that a sheet can be checked for? Or is it a design choice that sheets may not have such a state?

[saulpw]

It's an implicit design choice, I guess, because some sheets may never finish loading. We could make it so that if loading is False and rows is not UNLOADED then that must mean it's finished loading. But rather than providing a way to wait until loading is finished, I think it'd be better tofail and let people manually wait or cancel the loading.

[saulpw]

You're right, @midichef, this is safer. Thanks for addressing it.