Do concat reductions preserve array order or am I getting lucky?

[fclingen]

I've been playing with reductions that use CkReduction::concat to compile results from a large number of smaller worker chares. In my experimentation, I'm finding that the results data I get back is always in the same order as the indices of the chare array (so chare with index 0 has results in index 0 of the array passed to the callback, etc...). This happens even if I use 'sleep()' inside the chares so I KNOW which order they're going to finish in.

This is not what I would have expected. Rather, I would expect that I would not be able to count on the order of the results in the message passed to the callback, and I would need to index and sort those results before I do anything with them. It makes my life easier if I don't need to do that, but I can't find anyplace that says CkReduction::concat results have any order to them

[fclingen]

I was just getting lucky. Messing around with parameters more (especially number of processors) made the results more arbitrary in their return order.

[stwhite91]

concat makes no guarantees on ordering because doing otherwise would impose performance overheads on those kinds of reductions. In Charm++ the contributors to reductions are migratable, so the natural index ordering of contributors is not guaranteed to match their ordering across the physical PEs and Nodes of the system. This means that imposing an ordering would in general require extra communication.

If you do want to know who has contributed each subarray of a reduction, you can use a 'set' reducer, or else a 'tuple' reduction which is composed of two 'concat' reductions (one for the actual data contributions, one for the index of the contributor-- the two contributions are guaranteed to stay in the same order as each other).

[fclingen]

Erm, my plan was to define a structure that has an 'index' member and a 'data' member and use concat on that, iterating on indices in the array returned from the in the callback to get everything copied to the right place.

Because I'm working on f90charm, tuple reductions and set reductions make me leery because they rely more on passing C++ objects around which takes some setup to do through Fortran. Further, my impression of set reductions was that they're "concat, but for data you don't know the size of at compile time"?

Also, even if I was working in C++ instead of Fortran I'm not sure what the advantage is of using tuple reductions instead of passing structs around?

[sanjaykale]

Frank, How is it going? We have so few f90charm users that I worry it will get rusty. Please let us know if you encounter difficulties or features that are lacking. And if there successes and results, we'd love to know about them.

[fclingen]

Sorry for the long time between replies. I appreciate the support.

Currently the project I was using f90charm on is on pause, but it should hopefully resume in the next week or two. Last time I was working on it, I was having memory error issues that I think stem from common blocks being present in the code I am trying to retrofit with Charm++, but I didn't get enough time to investigate and determine if that's really the problem before the pause happened.