-
Notifications
You must be signed in to change notification settings - Fork 21.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Get the error: AttributeError: Can't pickle local object 'convert_frame.<locals>._convert_frame' #93470
Comments
torch.compile does not emit a model in the way you expect it to, and I think maybe some new documentation has led you astray. If you intend to pickle the exported model, give export a try. See the export section of https://pytorch.org/get-started/pytorch-2.0/ |
it sounds like @fladventurerob is trying to compile the model after loading it, not before exporting it. however, @fladventurerob it would be helpful if you can provide a runnable script for us to look at, rather than just a description. |
it might be because of multiprocess compile.
from torch._inductor import config
config.compile_threads = 1
Does the error reproduce? |
You are correct. The model exists already. I was loading it into a forward test script. Based upon the documentation I was assuming this line was needed to use an existing model, rather than to export the model. |
@fladventurerob any update on whether compile_threads=1 helps, or are you able to provide a repro script for us? |
I had the same issue -- setting |
For what it's worth, I'm using I haven't followed the changes made very closely recently, but I did see this merged today (though perhaps unrelated): triton-lang/triton#1133. |
I would not close this issue. It randomly appears when you are going to torch save a model that needs a lot of time to be trained, which means that nothing is saved. Could you update then |
I believe this one will be fixed by #101651 when it lands |
I met the same bug, here is the simplest test case I can offer:
The error is:
Maybe this information helps :) |
I'm having a similar issue, but with dataloaders with num_workers > 0. (In particular, when using spawn instead of fork, I think this is probably cause it copies the entire environment) |
#101107 seems relevant. |
|
Hello! I encounter this issue while compiling a transformation used within the multi-processing context of the data loader. Even though I intend for the transformation to run on the CPU, the GPU is detected leading to the following error:
Despite attempting to investigate the issue by commenting out calls to In an effort to resolve this error, I followed the recommendation and utilized spawn as the multiprocessing context for the
While preparing a pull request, I attempted to move the definition of
|
So unassigning myself from this because i couldn't justify spending more time, I got stuck with getting 3 final tests to pass here #101651 The core idea is just to wrap some functions into classes so they become picklable |
@anijain2305 to provide update |
Possible partial fix: |
AttributeError: Can't pickle local object 'TrainAugmentation.init..'. Hi I got this error, and i actually don't know how to fix. |
@anthai0908 qfgaohao/pytorch-ssd#71 seems more relevant to you. I think your comment is really long and unrelated. It might be nice to remove it as to not clutter this issue. |
thanks @ringohoffman. After training and exporting to onnx, I have one question, is it possible to deploy inferenceon python 3.6.9 with onnx format? |
馃悰 Describe the bug
When adding the line:
model = torch.compile(model)
after loading the model, this error occurs. When removing the line, the script functions as intended.Error logs
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'convert_frame.._convert_frame'
Minified repro
No response
cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @Xia-Weiwen @ipiszy @soumith @ngimel
The text was updated successfully, but these errors were encountered: