Support *Local methods in Collections API

[radekmie]

Background

Currently, Meteor's Collections API has two modes: synchronous one (fetch, insert, etc.) and asynchronous one (fetchAsync, insertAsync, etc.). Their detailed behavior in different Meteor versions is described below.

• Synchronous (fetch, insert, etc.):
  • v2, client: executes synchronously, returns synchronously.
  • v2, server: executes asynchronously, returns synchronously (once the Fiber returns).
  • v3, client: same as *Async.
  • v3, server: same as *Async.
• Asynchronous (fetchAsync, insertAsync, etc.):
  • v2, client: executes synchronously, returns asynchronously (wrapped in a Promise).
  • v2, server: executes asynchronously, returns asynchronously (wrapped in a Promise; once the Fiber returns).
  • v3, client: same as v2.
  • v3, server: executes asynchronously, returns asynchronously.

Now, since Meteor v3 cannot support synchronous database operations on the server (#11505, https://radekmie.dev/blog/on-the-road-to-fibers-free-meteor/), we have to make sure the entire ecosystem is ready for that. View libraries were already taken care of (meteor/react-packages#360, meteor/react-packages#385, meteor/react-packages#387, meteor/blaze#412, meteor/blaze#413), but even if these can work with async APIs, it comes with some drawbacks - it requires a lot of changes in the way our apps handle the data.

Proposal

In #12592 (comment), @Grubba27 has proposed *Local methods to keep the synchronous API around. It'd be available only for the LocalCollection instances, i.e., on the client, when using LocalCollection directly, and even on the server, if you connect to a DDP server there. In other words: it'd allow you to use Minimongo synchronously.

• Local (fetchLocal, insertLocal, etc.):
  • v2, LocalCollection: executes synchronously, returns synchronously.
  • v2, non-LocalCollection: ???.
  • v3, LocalCollection: executes synchronously, returns synchronously.
  • v3, non-LocalCollection: ???.

As you see, we'd need to fill the above gaps. I see a couple of options, each with some drawbacks:

1. Don't define them on non-LocalCollections at all.
   • Impossible to handle in TypeScript easily, as Mongo.Collection is isomorphic and can be either underneath.
2. Define them on non-LocalCollections and throw when called.
   • Runtime errors.
3. Define them on non-LocalCollections and return some dummy values.
   • Unexpected behavior, as operations are effectivelly no-ops.

I personally would vote for 2. It's easy to create an ESLint plugin that allows *Local methods to be executed only in certain paths (e.g., containing client), and it prevents any unexpected behavior - if it gets called on a non-LocalCollection, there's an error immediately.

Problems

While all read operations are trivial (data is there, just return it), I have a problem with handling write operations. Similarly to what we have in Meteor now, if you execute a write operation on the client, it may fail later without you knowing (you can react in the callback, but that's it).

That's why I propose to start off small, only with read operations first. That'd be:

• collection.findOneLocal.
• collection.countDocumentsLocal.
• collection.estimatedDocumentCountLocal.
• cursor.fetchLocal.

Note that I skipped a few methods:

• cursor.count as it's deprecated since 2.9
• cursor.forEach, cursor.map, and cursor[Symbol.iterator] as these have a trivial replacement, i.e., cursor.fetchLocal().*.
• cursor.observe and cursor.observeChanges as they remained synchronous (Release 3.0 observe made back to sync #12573).

[Julusian]

I think that 1 could be better handled with a small api change to how these collections are constructed.
It could be done non-breaking, by adding the ability to do new Mongo.LocalCollection() which will return the LocalCollection type. For anyone not using typescript or not needing the *Local properties, they could continue to use new Mongo.Collection(null)
This LocalCollection type would extend Collection, so anywhere that accepts Collection would also accept LocalCollection.

Ideally, LocalCollection should be a class that extends the Collection class so that doing myCollection instanceof LocalCollection behaves correctly, to allow for runtime differentiating between the two.

[radekmie]

That's also an option, though rather complicated, as right now Mongo.Collection is isomorphic and has a _collection field which is either LocalCollection or driver's Collection object. But sure, definitely doable.

My only problem is that it looses the ease of Mongo.Collection used in shared code, as with that, you'd have to call Mongo.Collection on the server and Mongo.LocalCollection on the client. It's also a little misleading, as the latter would technically be more like a "remote collection", i.e., one with local data but synced from the server.

[Julusian]

you'd have to call Mongo.Collection on the server and Mongo.LocalCollection on the client

You'd only have to if you want the correct typings.
I admit it does get a bit messy, but you could define the collections in lib, and then reexport them in client after casting them to LocalCollection (or whatever this should be called)

At work we have already split the collection definitions up to be defined once in server and once in lib (ideally this should be client not lib, but there is some shared code) so that we can get them strongly typed and differently async/sync. We also have classes wrapping the mongo collections, to make some adjustments so probably arent the best case to consider.

[Julusian]

It's easy to create an ESLint plugin that allows *Local

I don't think a good eslint rule will be that easy to write. A naive one will be, but that will catch any methods named *Local, not just these collection ones.
And using *Local on the server side is valid in some scenarios, which this rule will not consider.

For this second reason, my codebase would not be able to use this rule.

[radekmie]

Of course. I think most applications don't have a findOneLocal or countDocumentsLocal methods in their codebase, but it wouldn't cover all cases.

[jamauro]

I like the idea of having a synchronous way to read Minimongo data on the client since the async api seems unnecessary here. In an ideal world, I wouldn't need to specify *Local though. It would be nice if Meteor abstracted that away from me even if under the hood it's using *Local.

I think starting with read operations makes sense. In fact, that may be all that is needed. I'm not sure I understand the use case of having a way to write synchronously to a LocalCollection -- assuming that the async api supports Optimistic UI for write operations inside a Method.

[radekmie]

Yeah, that'd be ideal. However, that would break isomorphism. At the same time, Blaze or React integrations can both operate on cursors, so you don't need to think about it there.