Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow random arrays of arbitrary size / speed up random arrays #1448

Open
mrfh92 opened this issue Apr 19, 2024 · 3 comments · May be fixed by #1508
Open

Allow random arrays of arbitrary size / speed up random arrays #1448

mrfh92 opened this issue Apr 19, 2024 · 3 comments · May be fixed by #1508
Assignees
Labels
enhancement New feature or request ESAPCA relevant for the ESA-funded project "ESAPCA" random

Comments

@mrfh92
Copy link
Collaborator

mrfh92 commented Apr 19, 2024

The current very elaborate implementation of our random generator and random arrays suffers from two drawbacks:

  • the maximum number of entries is quite low (~maxint32 / maxint64 for the respective data types)
  • it is comparatively slow.

I suggest to consider something like the following:

global_seed = <...> 
local_seed = global_seed + ht.MPI_WORLD.rank 
local_rand = torch.rand(local_shape, ...) # with local_seed as seed 
global_array = ht.array(local_rand, ...) 

To discuss:

  • does this destroy the "quality" of the resulting pseudo-random array? (I know, as a trained mathematician I should be able to answer that question, but at the moment I can't)
  • even if the quality might be lower in the end, I suggest to add this version of random arrays, e.g., with a keyword argument quick_and_dirty=True or something like this
@mrfh92 mrfh92 added the enhancement New feature or request label Apr 19, 2024
@mrfh92 mrfh92 self-assigned this Apr 19, 2024
Copy link
Contributor

Branch features/1448-random_arrays_of_arbitrary_size created!

@mrfh92 mrfh92 changed the title random arrays of arbitrary size Allow random arrays of arbitrary size / speed up random arrays Apr 22, 2024
@mrfh92
Copy link
Collaborator Author

mrfh92 commented Apr 22, 2024

todo: look up how other libraries (dask, jax?) solve this problem

@mrfh92
Copy link
Collaborator Author

mrfh92 commented Apr 22, 2024

@Markus-Goetz what do you think about the suggested workaround?

@mrfh92 mrfh92 added ESAPCA relevant for the ESA-funded project "ESAPCA" random labels Apr 22, 2024
@mrfh92 mrfh92 linked a pull request Jun 3, 2024 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ESAPCA relevant for the ESA-funded project "ESAPCA" random
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant