The case for globals

[shikokuchuo]

As posed by @HenrikBengtsson in #45.

I was sitting in the SatRdays conference in London this afternoon. And I realised that there are potentially a lot of data scientists working with scripts containing variables and functions, all thrown together in the global environment.

And the situation Henrik pointed out, which I sort of dismissed at the time as a case of 'user error' might be more common than I had anticipated.

As #54 implements name = value pairs to be passed via .args, this frees up ... to be used for another purpose i.e. to pass globals (explicitly assigned to .Globalenv in the new process).

The advantage is that I think this is going to be mostly back compatible with existing code as globals remain on the search path.

An added bonus is that the ... is on the same level semantically as .expr in the function, and this also reflects their relative positions in the new process.

@wlandau this might allow you in crew to map data to .args and globals to ..., potentially saving you a layer?

I'm not committed to implementing this, just toying with the idea. Welcome any thoughts / comments.

[wlandau]

I agree it is good to explore this. If mirai supports a way to pass globals directly, then I will use it in crew. However, the simplification would be minor on my end. I am fine either way. And I do still prefer to support separate data and globals arguments in the crew interface in order to make the distinction as clear and obvious as possible.

Currently, crew uses ... for local variables. If mirai switched ... to globals instead of local variables, there would be edge cases in crew until I can release another version (e.g. if a user of crew defines a global object called data in the globals list which supersedes the data argument that crew passes to ... in mirai()). And where possible I would prefer to update crew on CRAN at most once a month in order to respect their guidelines.

[shikokuchuo]

I am committed to being able to have different local and global variables, so we're on the same page there.

[wlandau]

Yeah, for practical purposes, it's a minor point for me. On this topic, I am much more concerned about potentially breaking changes.

But if you ask my general opinion about designing interfaces, I prefer formal arguments over informal ones, long informative names over abbreviations, and as many arguments/functions as needed to create opportunities for documentation and to make each feature clear and consistent. I believe the complexity of a package comes from the underlying functionality, and that a good interface is right-sized for it, even if overloading would technically be possible. I also think the wishes of direct end users which call mirai in informal workflows may be different from those of package developers like Henrik and I who call mirai from other packages.

[shikokuchuo]

I would not as a matter of principle intentionally break crew so any change would be after you had moved to '.args'. But I am also cautious for my own users. This was only an idea as I thought it would be unlikely to break existing code, but who really knows...

I have taken the safest course of action for now, which is to beef up the examples!

I have been reluctant to open up new interfaces as .expr is so flexible itself. Someone asked for a .file interface some time ago and I pointed out you could simply call source() in your expression. In the same vein, .globals still feels somewhat arbitrary to me.

[shikokuchuo]

I've moved away from the idea of using ... for globals. This does not seem correct as a default.

I've explicitly added in the docs that evaluation does not occur in the global environment. Along with the example of how to assign variables to the global environment if you need to, I think is about as much as I can do.

Adding a .globals argument also doesn't seem right, as it potentially encourages the 'wrong' behaviour - it becomes convenient to always pass variables via .globals rather than .args as it 'just works'. I suppose we have to be a little bit opinionated on this matter :)

[wlandau]

Sounds very sensible, an opinionated nudge in the right direction.

[HenrikBengtsson]

Hello. I'm not sure if it makes a difference, but I use the term "global variable" in the traditional programming sense. For example, a is a global environment of g() in:

f <- function(x) {
  a <- 42
  g <- function(z) {
    a + z
  }
  g(x)
}

Thus, by this definition, a global variable does not have to live in the globalenv() environment.

From my perspective, I don't think there's a need to be able to assign something to the globalenv() of the worker. What argument .args does right now is sufficient for my needs in the future framework.

PS. Being able to properly reconstruct all the frames on a parallel worker, with just enough variables to resolve the expression, is really tricky. Things like packages being attached at run-time when the expression is evaluated, complicates it further.

[shikokuchuo]

Thanks @HenrikBengtsson for the clarification. That makes total sense now. Your insights are always very welcome!

[shikokuchuo]

After further feedback, practicality trumps perceived notions of 'correctness' and my original idea is implemented in 647e65d

[wlandau]

Does it need to be just one or the other? This is why crew controllers have separate arguments for globals and local data in push().

[shikokuchuo]

It's both - '...' maps to globals, '.args' maps to 'data'.

[wlandau]

Ah, okay. That solves that problem. I will switch crew to using .args.