Loading large dataset of images

[jitendersaini]

This is really a great application and I've enhanced it with integrating the google photos apis just to play the slideshow from cloud directly. But I really have to see the feasibility of loading my all photos in this as this might last forever to load and build the slideshow for my 7000 images. Any alternate to make it fast?

[peterdolenc]

Hello, thanks for the nice words.
Is your problem specific to the images that are being loaded from the google cloud?

I also have a similar amount of pictures and the frame takes 15-20 mins to process the images when rebooted. But that hardly ever happens so it's not an issue for me.
One way you could help your frame to process the images faster is by resizing them upfront with some image resizer software. Constrain their dimensions to dimensions of the screen you're using. It makes a big difference.

[jitendersaini]

Hello, Thanks for your prompt response.
I don't have a challenge to pull images from gphotos because that is a separate program I wrappered on it and it's working and populating all the 7000 URLS in a python list, but when rpy-frame is processing those 7000 images it's taking huge amount of time, I tried process 100 images first and on my Macbook pro laptop it's taking around 10 minutes and on RPI3 around 25-30 minutes. Actually I was thinking to make it possible that instead of process all the images in one go first we pick around 10 images process them and start the presentation and in the background it keeps on processing rest of the 6990 images and keeps on pushing the processed images on to the Queue. Actually I've achieved this task in Some other program that is based on python Tkinter but later on I found rpy-frame is too cool and decided to use it, but implementing the same background processing logic is little complex as there is very much use of Threading in this, or it will really be cool feature if we can give a flag in settings.py like NO_OF_IMAGES_TO_START_SLIDE_SHOW_AFTER_PROCESSING=10. Do you see if this is feasible in rpy?
Thanks again

[peterdolenc]

I think this could be implemented even without any additional settings and it's definitely very doable.
As soon as you have one image processed properly you could create a sequence with just that image. And after that you'd create a sequence with the images available at that time etc.
The app is already multi threaded and inter-thread communication is already in place where needed.

The drawback or the biggest challenge would probably be that on startup you would always see the same image(s) (ie. the first ones in the directory). So a good solution would also randomize the initial processing to avoid that.

[jitendersaini]

Thanks again for the reply. Yes you're absolutely right that I always see the same images those are loaded first even though Queue is keeps on pushing with the new images. I really need to check if some workaround for this. Below is the code snippet that I've made a temporary change to test it with gphotos albums
In file_loader.py

     def get_files_from_directory(path: str) -> List[str]:
        files = []

        files, albums_ids = get_google_photos()        
        th1 = threading.Thread(target=get_next_page_content, args=(files, albums_ids))
        th1.start()

        # for file in os.listdir(path):
        #     file_path = os.path.join(path, file)
        #     file_mode = os.stat(file_path)[stat.ST_MODE]
        #     if stat.S_ISDIR(file_mode):
        #         files.extend(FileLoader.get_files_from_directory(file_path))
        #     elif stat.S_ISREG(file_mode):
        #         _, ext = os.path.splitext(file)
        #         if ext.lower() in ['.png', '.jpg', '.jpeg', '.gif', '.bmp']:
        #             files.append(file_path)
        return files

get_google_photos is loading the first batch of 10 files and get_next_page_content is loading and pushing rest of the 6990 images into the files object.

Thank you for your followup