Data Residency

[RobertLowe]

It would be great if Meteor/Galaxy could take advantage of Data Residency.

Atlas introduced "Global Clusters" which allow you to specify where you would like data to reside. https://docs.atlas.mongodb.com/global-clusters/

Unique challenges:

• Adding location property to all or some? records
• Handling index changes, unique indexes no longer operate the same

Opening for discussion and interest

[StorytellerCZ]

Wouldn't this be more of a Mongo driver feature?

[RobertLowe]

Not exactly because the account system contains personal data, and unique indexes operate differently in a global cluster, there's some discover to be done in that regards

[StorytellerCZ]

OK, I'll read up a bit on it.

In the meantime from what you have written, is the first step handling the option to write the location property to the users collections?

As for the index handling, I will read up on it, but wouldn't that be something that MongoDB handles unless it is related to the generation of _id? Or is there something that we will have to implement from the MongoDB driver?

[RobertLowe]

Yes, location to users, the sharding key has to be: location+_id, this could probably be handled by adding METEOR_LOCATION_KEY global env var

Uniqueness will be an issue for emails, not so much _ids because of how Meteor doesn't use Mongo IDs but its own Random.id()

[StorytellerCZ]

I don't think that env variable is necessarily the correct way. For example a user that registers in Asia, but selects that they are from EU, which would mean that their data need to be stored in the EU region (according to law). I think the env variable could be default and automatically added on Galaxy, but there probably should be some function that would pass in the user info and then it should return the location to be applied to the user records.

[filipenevola]

This is a nice feature from MongoDB.

I already created a PoC for a few customers on Galaxy for them to be able to run multiple regions using the same domain, this is possible without new features in Galaxy using services like Route 53 where you can map different CNAME records for different source regions.

But I always mention to customers that the database would still be a problem as they would probably need a single database somewhere at least in the cases where the applications share data world wide, this wouldn't be a big deal for application that are multi-tenant and each tenant is always using from a specific region.

Now with this feature from MongoDB this single database problem would be solved.

I believe the point of this discussion, sorry if I misunderstood @RobertLowe, is to think about the places where Meteor core packages handle the data for developers, like accounts, to see how many changes we would need to do to support this use-case. Am I right?

A follow up question to you @RobertLowe, are you trying to use this MongoDB feature right now in your app or this discussion here is more to talk about the impact?

It would be nice to have someone implementing this with Meteor and MongoDB because we would have a real case to battle test this scenario.

I'll keep my eyes open for tickets in this direction from Galaxy customers.

[RobertLowe]

Now with this feature from MongoDB this single database problem would be solved.

Correct!

I believe the point of this discussion, sorry if I misunderstood @RobertLowe, is to think about the places where Meteor core packages handle the data for developers, like accounts, to see how many changes we would need to do to support this use-case. Am I right?

Correct! This is precisely what I'd like to begin.

A follow up question to you @RobertLowe, are you trying to use this MongoDB feature right now in your app or this discussion here is more to talk about the impact?

We are currently migrating functionality to utilize this feature. We've decided rather than auto-assigning location based on region (which is a great way to handle the case), as a part of sign-up / preferences, the user will be able to select their data-residency location. As setting or changing the location field MongoDB will handle migrating the data to the according region.

[filipenevola]

This is great, so you are the perfect candidate to guide us and also contribute to our changes in this direction.

Besides the user collection, are there other collections or packages that you believe would be affected?

[RobertLowe]

Off the top of my head I think the only internal package of concern is the accounts package aka users.

Speaking of packages though, we may want to plan how they can support data-residency, and for end users. Maybe a way to force packages to apply location property according to some configuration.

Also thinking about collections and configuration, we could expand the options of new Mongo.Collection(name, [options]) to include a hook: residency: (recordToBeStoredUpdated)=> ...

Something like:

let todos = new Meteor.Collection('todos', {
  residency: (record) => {
    // Store Todos based on the users' country
    let user = user.findOne(record.userId) // perhaps we could allow `Meteor.user()` here as well
    // America, or UK (default country)
    return user.country == "America" ? Meteor.Collection.US_Residency : Meteor.Collection.UK_Residency 
  }
}

[filipenevola]

I believe you could achieve this without changes in core packages.

For example, usually I create my collections using the package quave:collections and then I can provide composers. One composer could perform this residency transformation suggested by you.

[RobertLowe]

I believe the changes are worth it considering the importance to modern businesses with growing regulations residency is becoming an ever important part of data-governance. Such is supported by trends:

I agree, there are a number of libraries and strategies like quave:collections which let you utilize a hook to handle such a case. However plenty of packages don't export their internals, so you're unable to make such changes. What I think we should consider is enabling developers to force a package's collections to handle some kind of residency, this way the whole ecosystem would in effect over night be able to support residency.

For ideation consider the pseudo code:

Package['somePackage'].setResidency((record, collection?) => ...)
// or for all packages:
Package.setResidency((record, collection?, package?) => ...)
// or for everything:
Mongo.setResidency(...)

This could supply/override Mongo.Collection and merge the residency hook on new Mongo.Collection(...)

I think this feature would be value enough to consider for core.