[WASI-NN] neural_speed: add backend structure #3303

grorge123 · 2024-03-25T07:14:08Z

No description provided.

juntao · 2024-03-25T07:14:11Z

Hello, I am a code review bot on flows.network. Here are my reviews of code commits in this PR.

codecov · 2024-03-25T12:05:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.84%. Comparing base (5a65509) to head (107c232).
Report is 3 commits behind head on master.

❗ Current head 107c232 differs from pull request most recent head c4b523c

Please upload reports for the commit c4b523c to get more accurate results.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3303   +/-   ##
=======================================
  Coverage   79.83%   79.84%           
=======================================
  Files         253      253           
  Lines       34948    34948           
  Branches     6138     6122   -16     
=======================================
+ Hits        27902    27905    +3     
+ Misses       5625     5623    -2     
+ Partials     1421     1420    -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hydai · 2024-05-31T07:18:30Z

plugins/wasi_nn/CMakeLists.txt

@@ -149,6 +149,64 @@ if(BACKEND STREQUAL "ggml")
 endif()
 endif()

+if(BACKEND STREQUAL "neuralspeed")
+ find_package(simdjson QUIET)


We changed the way to find the simdjson package, please use the new way like this pr: #3426

I have changed. Please check, thank you.

hydai · 2024-05-31T07:19:43Z

plugins/wasi_nn/neuralspeed.cpp

+#endif
+void printImformation(Graph &GraphRef, Context &CxtRef) {
+ spdlog::info(
+ "[WASI-NN][Info] Neural speed backend: Number of input tokens: {}"sv,


You don't need to add [Info] here, because the spdlog::info will do this for you.

I have changed. Please check, thank you.

hydai · 2024-05-31T07:20:44Z

plugins/wasi_nn/neuralspeed.cpp

+ "[WASI-NN][Debug] Neural speed: Model path not found in nn-preload, "
+ "write model into a tmpfile."sv);
+ }
+ // TODO: pass the model directly to ggml


I would like to know the details of this TODO item.

It refers to ggml backend. Although current implementation is enough. Should I remove it?

We can remove it.

I have changed. Please check, thank you.

hydai · 2024-05-31T07:21:37Z

plugins/wasi_nn/neuralspeed.cpp

+ auto &CxtRef = Env.NNContext[ContextId].get<Context>();
+ auto &GraphRef = Env.NNGraph[CxtRef.GraphId].get<Graph>();
+ if (!Py_IsInitialized()) {
+ spdlog::info(


We should use spdlog::error here.

I have changed. Please check, thank you.

hydai · 2024-05-31T07:22:39Z

plugins/wasi_nn/neuralspeed.cpp

+Expect<WASINN::ErrNo> compute(WasiNNEnvironment &Env,
+ uint32_t ContextId) noexcept {
+ if (!Py_IsInitialized()) {
+ spdlog::info(


We should use spdlog::error here. This is a fatal error.

I have changed. Please check, thank you.

hydai · 2024-05-31T07:23:25Z

plugins/wasi_nn/neuralspeed.cpp

+ "[WASI-NN] neural speed backend: Input transfer tensor failed."sv);
+ return WASINN::ErrNo::InvalidArgument;
+ }
+ // PyObject *GenerateArgs = PyTuple_Pack(1, LongTensor);


Why does this line comment out?

I have changed. Please check, thank you.

hydai · 2024-05-31T07:23:48Z

plugins/wasi_nn/neuralspeed.cpp

+ }
+ }
+ Py_DECREF(Result);
+ // Py_DECREF(GenerateArgs);


I have changed. Please check, thank you.

hydai · 2024-05-31T07:26:45Z

test/plugins/wasi_nn/CMakeLists.txt

+ elseif(BACKEND STREQUAL "neuralspeed")
+ message( STATUS "Download ML artifacts to ${CMAKE_CURRENT_BINARY_DIR}/wasinn_neural_speed_fixtures")
+ download(
+ https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_0.gguf


Do we really need to download this huge file if we would like to do the test?
Is it possible to have a small one instead?

I have changed. Please check, thank you.

hydai · 2024-05-31T07:29:57Z

test/plugins/wasi_nn/wasi_nn.cpp

+ uint32_t BuilderPtr = UINT32_C(0);
+ uint32_t LoadEntryPtr = UINT32_C(0);
+ uint32_t SetInputEntryPtr = UINT32_C(0);
+ uint32_t OutBoundPtr = UINT32_C(61000 * 65536);


Should we use UINT32_C(61000) * UINT32_C(65536)?

I have changed. Please check, thank you.

Signed-off-by: grorge <[email protected]>

hydai

And I think we should also add the workflow to build and upload the assets.

hydai · 2024-06-02T17:37:48Z

plugins/wasi_nn/neuralspeed.cpp

+ return ErrNo::InvalidEncoding;
+ }
+ if (Doc.at_key("model_type").error() == simdjson::SUCCESS) {
+ std::string_view model_type;


ModelType.

I have changed it.

Signed-off-by: grorge <[email protected]>

grorge123 requested a review from ibmibmibm as a code owner March 25, 2024 07:14

github-actions bot added c-Plugin An issue related to WasmEdge Plugin c-WASI-NN c-CMake labels Mar 25, 2024

grorge123 force-pushed the neural_speed branch 4 times, most recently from f916aeb to af7d519 Compare March 25, 2024 11:53

grorge123 requested a review from q82419 as a code owner March 25, 2024 11:53

github-actions bot added the c-Test An issue/PR to enhance the test suite label Mar 25, 2024

grorge123 force-pushed the neural_speed branch from af7d519 to fb9bdc7 Compare March 25, 2024 11:54

github-actions bot removed the c-Test An issue/PR to enhance the test suite label Mar 25, 2024

grorge123 marked this pull request as draft March 25, 2024 12:55

grorge123 marked this pull request as ready for review March 29, 2024 06:22

grorge123 changed the title ~~[WIP] [WASI-NN] neural_speed: add backend struct~~ [WASI-NN] neural_speed: add backend structure Mar 29, 2024

grorge123 requested a review from hydai as a code owner April 8, 2024 06:35

github-actions bot added the c-Test An issue/PR to enhance the test suite label Apr 8, 2024

grorge123 force-pushed the neural_speed branch from 2ad58a0 to 17f0157 Compare April 25, 2024 03:23

github-actions bot added c-CLI An issue related to WasmEdge CLI tools c-Installer c-CI labels Apr 25, 2024

grorge123 force-pushed the neural_speed branch from 17f0157 to 2ad58a0 Compare April 25, 2024 03:34

github-actions bot removed c-CLI An issue related to WasmEdge CLI tools c-Installer c-CI labels Apr 25, 2024

grorge123 force-pushed the neural_speed branch 3 times, most recently from a60159f to 984d13a Compare April 25, 2024 04:06

grorge123 force-pushed the neural_speed branch 5 times, most recently from ef72299 to 065fc95 Compare May 13, 2024 07:19

hydai added the v-0.14.1 label May 22, 2024

hydai requested changes May 31, 2024

View reviewed changes

grorge123 added 16 commits June 1, 2024 14:44

[WASI-NN] neural_speed: add backend struct

3bb5303

Signed-off-by: grorge <[email protected]>

test: add successful test

628c6d4

Signed-off-by: grorge <[email protected]>

feat: implement neural network backend

84859e8

Signed-off-by: grorge <[email protected]>

fix: neural backend and basic test

2e4d4ea

Signed-off-by: grorge <[email protected]>

fix: update pythonlib and share library

a4a6d6a

Signed-off-by: grorge <[email protected]>

feat: add setting model type

82ae414

Signed-off-by: grorge <[email protected]>

fix: python finalize deadlock

1d1c152

Signed-off-by: grorge <[email protected]>

test: add successful test

fa68cb9

Signed-off-by: grorge <[email protected]>

feat: update build neural speed cmake

e30bdaf

Signed-off-by: grorge <[email protected]>

feat: aneural speed backend ad unload function

405e4f8

Signed-off-by: grorge <[email protected]>

fix: simdjson not fund

f1a6851

Signed-off-by: grorge <[email protected]>

fix: remove finiSingle and set simdjson tags/v3.9.1

ff5826a

Signed-off-by: grorge <[email protected]>

feat: print run time

c973cda

Signed-off-by: grorge <[email protected]>

fix: change time to chrono

ca859d1

Signed-off-by: grorge <[email protected]>

fix: windows dlopen

37c4a65

Signed-off-by: grorge <[email protected]>

test: add more test

acc887c

Signed-off-by: grorge <[email protected]>

grorge123 force-pushed the neural_speed branch from 065fc95 to b98dbea Compare June 1, 2024 09:08

[WASI-NN] neural_speed: change simdjson and small model

25be440

Signed-off-by: grorge <[email protected]>

grorge123 force-pushed the neural_speed branch from b98dbea to 25be440 Compare June 1, 2024 09:12

hydai requested changes Jun 2, 2024

View reviewed changes

github-actions bot added the c-CI label Jun 3, 2024

[WASI-NN] neural_speed: Add build workflow

c4b523c

Signed-off-by: grorge <[email protected]>

grorge123 force-pushed the neural_speed branch from 107c232 to c4b523c Compare June 3, 2024 11:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WASI-NN] neural_speed: add backend structure #3303

[WASI-NN] neural_speed: add backend structure #3303

grorge123 commented Mar 25, 2024

juntao commented Mar 25, 2024 •

edited

codecov bot commented Mar 25, 2024 •

edited

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 May 31, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai May 31, 2024

grorge123 Jun 1, 2024

hydai left a comment

hydai Jun 2, 2024

grorge123 Jun 3, 2024

[WASI-NN] neural_speed: add backend structure #3303

Are you sure you want to change the base?

[WASI-NN] neural_speed: add backend structure #3303

Conversation

grorge123 commented Mar 25, 2024

juntao commented Mar 25, 2024 • edited

codecov bot commented Mar 25, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hydai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juntao commented Mar 25, 2024 •

edited

codecov bot commented Mar 25, 2024 •

edited