Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing - Progress bar not updating for starmap #1558

Open
5 of 6 tasks
jetlime opened this issue Mar 10, 2024 · 0 comments
Open
5 of 6 tasks

Multiprocessing - Progress bar not updating for starmap #1558

jetlime opened this issue Mar 10, 2024 · 0 comments

Comments

@jetlime
Copy link

jetlime commented Mar 10, 2024

  • I have marked all applicable categories:
    • exception-raising bug
    • visual output bug
  • I have visited the source website, and in particular
    read the known issues
  • I have searched through the issue tracker for duplicates
  • I have mentioned version numbers, operating system and
    environment, where applicable:

4.66.2 3.9.18 (main, Jan 4 2024, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] linux

I am performing multi-processing on a python dictionarry using the multiprocessing.starmap() method.

pool = multiprocessing.Pool()

with pool as p:
    with tqdm.tqdm(total=len(dict)) as pbar:
        for _ in p.starmap(func, dict.items()):
            pbar.update()    
            
pool.close()
pool.join()

However, this implementation does not update the progress bar. The bar is either empty during the complete execution of the program:

  0%|                                                                                            | 0/2414 [00:07<?, ?it/s]

Or it is complete, at the end of the execution:

100%|█████████████████████████████████████████████████████████████████████████████████| 2414/2414 [01:42<00:00, 23.53it/s]

I understand it is due to the fact that the starmap() method does not return an iterator. This code would work if using the imap() method though. While using starmap(), what are the possibilities?

@jetlime jetlime changed the title Multiprocessing - Progress bar not updating Multiprocessing - Progress bar not updating for starmap Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant