Allow to pass second iterable to tqdm() to display custom progress?

[JohannesWiesner]

• I have marked all applicable categories:
  • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand [StackOverflow#tqdm] is more appropriate)
  • new feature request
• I have visited the [source website], and in particular
  read the [known issues]
• I have searched through the [issue tracker] for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

4.66.1 3.12.0 | packaged by conda-forge | (main, Oct  3 2023, 08:43:22) [GCC 12.3.0] linux

Hi! I already posted a thread on SO but I think I have to specify my question further. I would like to separate the internal logic of my function from the optional progress output using tqdm. Specifically I would like to print the current amount of data copied if wished by the user. For that, it would be nice, if I could use tqdm(my_iterable) to "activate" the progress bar but then also pass a second iterable to it (cumulative file sizes) that is used to show the progress. Here's an example:

import os
import shutil
import time
from tqdm import tqdm
import numpy as np

###############################################################################
## Example Preparation 
###############################################################################

# start with a clean directory
if os.path.isdir('./src'):
    shutil.rmtree('./src')
if os.path.isdir('./dst'):
    shutil.rmtree('./dst')

# create test files with different sizes
os.makedirs('./src')

for i in range(10):
    with open(f"./src/{i}", 'wb') as bigfile:
        bigfile.seek(1048575 * i)
        bigfile.write(b'0')

# get list of filepaths and list of destination directories where these files
# should be copied to
files = [os.path.join('./src', file) for file in os.listdir('./src')]
dst_dirs = ['./dst' for i in range(len(files))]

# get cumulative sizes of files (in percentage)
filesizes = [os.path.getsize(file) for file in files]
bytes_total = sum(filesizes)
cumulative_sizes = list(np.cumsum(filesizes) / bytes_total * 100)

###############################################################################
## Actual Question
###############################################################################

# this function should NOT use tqdm internally (separate logic of function
# from displaying user information)
def copy_files(files,dst_dirs):
    '''Copy files based on a list of source filepaths and a corresponding
    list of destination directories.
    '''
    
    for dst_dir in set(dst_dirs):
        os.makedirs(dst_dir)
         
    for file,dst_dir in zip(files,dst_dirs):
        shutil.copy2(file,dst_dir)
        
        # just for dev purposes so we can actually see the output
        time.sleep(0.2)

# set verbosity
verbose = True

if verbose == True:
    # would be nice if we also could pass cumulative_sizes to tqdm() so it can 
    # display the current progress of copied data
    copy_files(tqdm(files,desc='Copy files'), dst_dirs)
elif verbose == False:
    copy_files(files, dst_dirs)

[xuancong84]

I think a more general solution is to pass in a lambda expression (or function) which takes the current item as the argument. For example,

for i in tqdm(range(100), display_current=lambda t:f'Item{t}'):
 # do task

@JohannesWiesner, it is too restrictive to pass in a list for displaying the current item because tqdm does allow passing in an iterator, in which case the actual label name of subsequent items might not be ready at the time when you call tqdm() function.

[JohannesWiesner]

@xuancong84 Will try that, thanks! What is display_current doing? Couldn't find any documentation on this

[xuancong84]

@JohannesWiesner It is my proposal, so not implemented yet. Would be glad if the developer can implement in this way.