Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using DuckDB with Deno #23656

Open
nshiab opened this issue May 2, 2024 · 20 comments
Open

Using DuckDB with Deno #23656

nshiab opened this issue May 2, 2024 · 20 comments
Labels
bug Something isn't working node compat node native extension related to the node-api (.node)

Comments

@nshiab
Copy link

nshiab commented May 2, 2024

Version: Deno 1.41.3

Hi!

I tried to use duckdb-node with Deno by running this index.ts.

import duckdb from "npm:duckdb";

const db = new duckdb.Database(":memory:");

But when running deno run -A index.ts, I get this error:

naelshiab test  $ deno run -A index.ts   
error: Uncaught (in promise) Error: Cannot find module '/Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/binding/duckdb.node'
Require stack:
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb.js
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb.js
    at Function.Module._resolveFilename (node:module:609:15)
    at Function.Module._load (node:module:487:27)
    at Module.require (node:module:662:19)
    at require (node:module:776:16)
    at Object.<anonymous> (file:///Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file:///Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:718:34)
    at Object.Module._extensions..js (node:module:732:10)
    at Module.load (node:module:643:32)
    at Function.Module._load (node:module:524:12)

Is it possible to use DuckDB with Deno?

Thank you very much for your great work.

@bartlomieju
Copy link
Member

This is because we currently don't support "lifecycle hooks" when installing npm dependencies (eg. postinstall script). We have a PR that implements that that's WIP: #23558.

You can work around this for now if you use "bring your own node modules" - by installing using npm/yarn/pnpm.

@bartlomieju bartlomieju added bug Something isn't working node compat node native extension related to the node-api (.node) labels May 2, 2024
@nshiab
Copy link
Author

nshiab commented May 2, 2024

Thank you!

@MaximePawlakFr
Copy link

MaximePawlakFr commented May 6, 2024

Hi @nshiab , have you succeeded?

I tried but I got this error:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

My data/index.ts contains:

import duckdb from "duckdb";
console.log(duckdb)
const db = new duckdb.Database(":memory:");
console.log(db)

The steps are:

  • setup the project with npm: npm init
  • npm install duckdb
  • try to run: deno run --allow-all ./data/index.ts
error: Uncaught (in promise) Error: Cannot find module 'MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/binding/duckdb.node'
Require stack:
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb.js
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb.js
    at Function.Module._resolveFilename (node:module:622:15)
    at Function.Module._load (node:module:500:27)
    at Module.require (node:module:675:19)
    at require (node:module:789:16)
    at Object.<anonymous> (file://MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file://MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:731:34)
    at Object.Module._extensions..js (node:module:745:10)
    at Module.load (node:module:656:32)
    at Function.Module._load (node:module:537:12)
  • then move to duckdb: cd node_modules/duckdb
  • install it: npm install
  • I can see the missing duckdb.node under node_modules > duckdb > lib > binding > duckdb.node
  • run again: deno run --allow-all ./data/index.ts
  • and finally get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

By adding RUST_BACKTRACE=1, I get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: core::result::Result<T,E>::unwrap
   4: _napi_set_element
   5: __ZNK4Napi6Object3SetIA11_cEEbjRKT_
   6: __ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE
   7: __ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE
   8: _napi_register_module_v1
   9: deno_napi::op_napi_open<NP>::v8_fn_ptr
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

And by adding RUST_BACKTRACE=full, I get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
stack backtrace:
   0:        0x1018f46b0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hade97c44b56fc870
   1:        0x100bcd6b0 - core::fmt::write::h81cbefbffc581dab
   2:        0x101901918 - std::io::Write::write_fmt::h125c60058ebfe43c
   3:        0x1018f4508 - std::sys_common::backtrace::print::hfa54be0dd0cf5860
   4:        0x10190838c - std::panicking::default_hook::{{closure}}::h4235e0929057f079
   5:        0x1019080d0 - std::panicking::default_hook::hcf67171e7c25be94
   6:        0x100b05c18 - deno::setup_panic_hook::{{closure}}::h71a800dcbf73da30
   7:        0x101908a2c - std::panicking::rust_panic_with_hook::h1767d40d669aa9fe
   8:        0x1018f49e8 - std::panicking::begin_panic_handler::{{closure}}::h83ff281d56dc913c
   9:        0x1018f4954 - std::sys_common::backtrace::__rust_end_short_backtrace::h2f399e8aa761a4f1
  10:        0x1019086b4 - _rust_begin_unwind
  11:        0x100bd78cc - core::panicking::panic_fmt::hc32404f2b732859f
  12:        0x100bd16f0 - core::result::unwrap_failed::h2ea3b6e22f1f6a7c
  13:        0x1007888a0 - core::result::Result<T,E>::unwrap::h2d955a3171e3fa0a
  14:        0x10078f060 - _napi_set_element
  15:        0x11a7719c0 - __ZNK4Napi6Object3SetIA11_cEEbjRKT_
  16:        0x11a7710a4 - __ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE
  17:        0x11a7728a0 - __ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE
  18:        0x11a771f38 - _napi_register_module_v1
  19:        0x101408960 - deno_napi::op_napi_open<NP>::v8_fn_ptr::h315a352275b5cff8

Have you managed to use duckdb in Deno?
What could I do?

Best

@MaximePawlakFr
Copy link

MaximePawlakFr commented May 6, 2024

--Update--

I've upgraded to deno 1.43.1 (was 1.40.4), and now I get this error:

error: Uncaught (in promise) Error: Error in native callback
    at Object.Module._extensions..node (node:module:704:20)
    at Module.load (node:module:597:32)
    at Function.Module._load (node:module:484:12)
    at Module.require (node:module:609:19)
    at require (node:module:715:16)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:659:34)
    at Object.Module._extensions..js (node:module:673:10)
    at Module.load (node:module:597:32)

Do you have any idea?

@nshiab
Copy link
Author

nshiab commented May 6, 2024

Hi @MaximePawlakFr!

For now, I am using node/npm. But @bartlomieju is working on it #23558.

@bartlomieju
Copy link
Member

It's an internal bug in Deno, but it seems easy enough to fix it to make example from DuckDB's page work. I'll try to have it done tonight.

@bartlomieju
Copy link
Member

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

@MaximePawlakFr
Copy link

Thank you Bartek.

I've just updated deno to deno 1.43.1+cbb78e1.

I tried again. I can load duckdb without the previous error.

But I can not run any request to duckdb. I get a segmentation fault.

Here is my code:

import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 
console.log(db);

Deno.serve(async (req) => {

    const request =
      "SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;";
    console.log(request);

    const result = await new Promise((resolve, reject) => {
      db.all(request, function (err, res) {
        console.log(res);
        if (err) {
          console.warn(err);
          reject(err);
        }
        resolve(res);
      });
    });

    const res = JSON.stringify(
      result,
      (key, value) => typeof value === "bigint" ? value.toString() : value, // return everything else unchanged
    );

    return new Response(
      res,
      {
        headers: { "Content-Type": "application/json" },
      },
    );
});

I run deno run --allow-all index.ts

I see in my terminal:

Database {}
Listening on http://localhost:8000/
SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;
zsh: segmentation fault  deno run --allow-all index.ts

It looks like something went wrong with duckdb. Any idea what?

Regards

@bartlomieju
Copy link
Member

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

@nshiab
Copy link
Author

nshiab commented May 7, 2024

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

Unfortunately, I am on a M1 Mac, so I can't install canary versions, from what I gathered.

@bartlomieju
Copy link
Member

@nshiab you can, we've been shipping canary version for ARM Macs for a couple months now :)

@nshiab
Copy link
Author

nshiab commented May 7, 2024

@bartlomieju What am I doing wrong? 😢

Screenshot 2024-05-07 at 9 04 47 AM

Thank you so much for your help!

@bartlomieju
Copy link
Member

It appears that deno binary you have locally on your system was installed via homebrew. You should remove it using homebrew and then install it again using the script, then add it to you PATH.

@nshiab
Copy link
Author

nshiab commented May 7, 2024

You are right. Stupid mistake on my end. Sorry.

Unfortunately, it doesn't seem to work on my laptop. Should I do something differently?

Screenshot 2024-05-07 at 10 32 33 AM

@bartlomieju
Copy link
Member

@nshiab the problem with postinstall scripts is not solved - you still need to use npm install and use DENO_FUTURE=1 env var, I fixed the problem related to actual execution of DuckDB.

@nshiab
Copy link
Author

nshiab commented May 7, 2024

Hmmm. I tried again but I am still doing something wrong I guess. It's probably best if I wait for postinstall scripts to work.

Screenshot 2024-05-07 at 4 00 46 PM

@bartlomieju
Copy link
Member

@nshiab you need to specify DENO_FUTURE env var in your terminal when you run the deno command:

# bash 
DENO_FUTURE=1 deno run -A index.ts
# fish
env DENO_DUTURE=1 deno run -A index.ts

@nshiab
Copy link
Author

nshiab commented May 7, 2024

IT WORKS! Amazing! Thanks for your patience @bartlomieju!

But... if I actually run a query, it fails. This is an example from the duckdb repo.

Screenshot 2024-05-07 at 4 33 24 PM

@bartlomieju
Copy link
Member

Thanks! I don't get a segfault, but I'm getting a panic like so:

import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 


const con = db.connect();

con.all("SELECT 42 AS fortytwo", function (err, res) {
  if (err) {
    console.warn(err);
  }
  console.log(res[0].fortytwo);
})
env DENO_FUTURE=1 deno run -A index.ts

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.43.1+998036b
Args: ["deno", "run", "-A", "index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2348:63:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'll try to debug this one as well.

@MaximePawlakFr
Copy link

MaximePawlakFr commented May 7, 2024

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

I've just update to the last canary: deno 1.43.1+998036b
I still get the same segmentation fault.

Keep me in the loop if you make any advance.

Thanks a lot for what you tried.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working node compat node native extension related to the node-api (.node)
Projects
None yet
Development

No branches or pull requests

3 participants