-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"Killed" during dataset convert from COCO to YOLO format #267
Comments
Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap. |
How many images are in your dataset? How large are the images? This error is the Out of Memory (OOM) killer on your machine acting to ensure the Python process doesn't take up too much RAM and cause instability on your system. This suggests your system isn't able to store all of the images in your dataset in memory, which is required to convert the datasets. |
i have about 6000 images (1024x1024). So if it is OOM problem, any suggestions how can i get around this problem? |
Hi, @DovydasPociusDroneTeam 👋🏻 This is interesting. So the script died, but the output datasets got saved anyway? Would love to learn more. |
Hi @DovydasPociusDroneTeam 👋 , we can dig deeper into the process to check for memory leakage but that will take some time. But answer to another quesiton about class names are rearranged, I might have an idea. @SkalskiP this is due to sorting class based on alphabetic order check here. |
@hardikdava I'm not sure. We got the input dataset. We divided that dataset into two parts. Saved both parts in YOLO format. Both Output datasets have different class orders. Do I understand the problem correctly? Is the order different between input and output datasets? Or between both output datasets? @hardikdava It is somehow a related topic. I think in the future, we should migrate |
@SkalskiP The changing of |
i didn't get output from one 6000 images dataset. So i tried this dataset split in to 3 separates datasets:
i did
and for every separate dataset with 2000 images i ran script from_coco().as_yolo() and and i was able to get results without error, but then i checked every output yaml file and saw "names" array was not same. |
@DovydasPociusDroneTeam, thanks a lot for helping us to understand what's happening. Could you help us a bit more and check the Please paste |
You are right! In my coco_part1.json and coco_part2.json categories are not in the same sequence! Okeyy... So i used bad converter from LABELME to COCO (when splitted dataset to 3 separates), don't know why it mixed categories sequence.. Thank you for that info! |
@DovydasPociusDroneTeam 🔥 Awesome that we managed to get to the bottom of this problem.
We will need to introduce lazy loading of images to make that happen. It is on our roadmap. I'll pin this issue there to keep track of that problem. I'll close the issue for now. |
@SkalskiP Can't we just save the dataset in pandas dataset then retrieve it batch by batch? |
Hi @Killua7362 👋🏻 Could you elaborate? |
Hello @SkalskiP |
Hi @Killua7362 👋🏻 No worries. I'm happy to explain. For now, you will always load the whole dataset, but we are thinking about adding a generator option. |
Can I try adding that option if you don't mind? @SkalskiP |
Hi @Killua7362, there is already the issue and PR, but I didn't have time to review it yet. |
@SkalskiP i use this datasets with 1 label https://universe.roboflow.com/naumov-igor-segmentation/car-segmetarion: but i use this script coco2yolo,but i got 2 labeles
and the generated format doesn't seem right either |
Hi @lonngxiang 👋🏻 I'm happy to help out. I just responded to your issue. Let's move the conversation there. |
Search before asking
Bug
getting "Killed" error while converting dataset from coco to yolo (the code is given bellow):
i tried to split manually big dataset in smaller parts (3 parts) and then didn't get error, but in .YAML file i got different classes positions in "names" part
and
any suggestions? Thank you in advance!
Environment
Minimal Reproducible Example
Additional
No response
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: