-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ERROR: Overriding output data pointer allocated by memory plan is not allowed. #3528
Comments
E 00:00:00.214101 executorch:method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed. This is noisy logging. I 0x2 just means the output location was memory planned which isnt really an error. This logging will be fixed in a future release. @cccclai Im guessing the actual problem is with qnn but not sure |
I 00:00:00.217469 executorch:qnn_executor_runner.cpp:415] Model executed successfully. @sunqijie0350 I see this. Are you sure its not running correctly? |
Yes, I have a relatively high probability that I can determine that it ran wrong, and a successful run would have the number of generated tokens and runtime printed, but apparently not here. What's strange is that retrying gives two different results. The probability of the above log appearing during a retry is small, the probability of the below log appearing is higher: ~/DataDisk/qijie.sun/push_files$ adb shell "cd ${DEVICE_DIR} [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2 [WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded! [INFO] [Qnn ExecuTorch]: Running level=3 optimization. [WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols [ERROR] [Qnn ExecuTorch]: SSR Detected - You must free and recreate affected QNN API handles associated with deviceId 0 coreId 0 pdId 0 [ERROR] [Qnn ExecuTorch]: Transport.teardownLocked: qnn_transport_teardown failed 0x00000027 [ERROR] [Qnn ExecuTorch]: Transport.teardownLocked: qnn_close error 0x00000027, userCnt 0 [ERROR] [Qnn ExecuTorch]: Transport.teardownLocked failed, error 0x00000007 [WARNING] [Qnn ExecuTorch]: All internal QNN API handles invalidated [ERROR] [Qnn ExecuTorch]: Graph failed in execution with err 1007 [ERROR] [Qnn ExecuTorch]: qnn_graph_execute failed. Error 1007 |
The log of a successful run is as follows: I'm wondering if the error reported above affects this result output? |
[ERROR] [Qnn ExecuTorch]: qnn_graph_execute failed. Error 1007 @cccclai can u take a look |
This looks like a failure in fastrpc.
Does it always happen? It's usually caused by models larger than what Hexagon can afford... are you using llama2-7b or lager? |
Same error on Windows on ARM. I'm running dummy llama.
|
:~/DataDisk/qijie.sun/push_files$ adb shell "cd ${DEVICE_DIR}
&& export LD_LIBRARY_PATH=${DEVICE_DIR}
&& export ADSP_LIBRARY_PATH=${DEVICE_DIR}
&& ./qnn_executor_runner --model_path ./dummy_llama2_qnn.pte"
I 00:00:00.001473 executorch:qnn_executor_runner.cpp:131] Model file ./dummy_llama2_qnn.pte is loaded.
I 00:00:00.001617 executorch:qnn_executor_runner.cpp:140] Using method forward
I 00:00:00.001694 executorch:qnn_executor_runner.cpp:188] Setting up planned buffer 0, size 14016.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[WARNING] [Qnn ExecuTorch]: Initializing HtpProvider
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
I 00:00:00.213863 executorch:qnn_executor_runner.cpp:214] Method loaded.
E 00:00:00.214101 executorch:method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
E 00:00:00.214145 executorch:qnn_executor_runner.cpp:263] ignoring error from set_output_data_ptr(): 0x2
I 00:00:00.214171 executorch:qnn_executor_runner.cpp:266] Inputs prepared.
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
I 00:00:00.217469 executorch:qnn_executor_runner.cpp:415] Model executed successfully.
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[WARNING] [Qnn ExecuTorch]: sg_stubPtr is not null, skip loadRemoteSymbols
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[WARNING] [Qnn ExecuTorch]: qnnOpPackageManager: hexagon unload op package function pointer is nullptr!
[WARNING] [Qnn ExecuTorch]: Function not called, PrepareLib isn't loaded!
Normal running model should have the output of inference result and time, but here is obviously not, please consult your experts, what is wrong?
The text was updated successfully, but these errors were encountered: