Running large migration file fails

[unidesigner]

Describe the bug

Running a migration file of size 67MB fails with error below. I also see a debug error from surreal binary: WebSocket error: Error { inner: Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }) }. I wonder if this has to do with surrealdb/surrealdb#2965

To Reproduce

I can share the file if necessary.

Expected behavior

Run the migration.

Information

• SurrealDB version: 1.0.0-beta.9+20230402.5eafebd for linux on x86_64
• surrealdb-migration version: 1.0.0

Additional context

Could it be related to WEBSOCKET_MAX_FRAME_SIZE being only 16MB see here.

Error: 
   0: There was an error processing a remote WS request: IO error: Connection reset by peer (os error 104)
   1: There was an error processing a remote WS request: IO error: Connection reset by peer (os error 104)

Location:
   src/surrealdb.rs:156

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: color_eyre::config::EyreHook::into_eyre_hook::{{closure}}::hd7da39aef3605027
      at <unknown source file>:<unknown line>
   2: eyre::capture_handler::hb1303c31a6ee2ec0
      at <unknown source file>:<unknown line>
   3: eyre::error::<impl eyre::Report>::from_std::hced59842f21ef8b9
      at <unknown source file>:<unknown line>
   4: surrealdb_migrations::surrealdb::apply_in_transaction::{{closure}}::hca01a80ce835ab29
      at <unknown source file>:<unknown line>
   5: surrealdb_migrations::apply::main::{{closure}}::h8128f5e36f828d15
      at <unknown source file>:<unknown line>
   6: surrealdb_migrations::main::{{closure}}::hc0a43bb13bf60e85
      at <unknown source file>:<unknown line>
   7: tokio::runtime::park::CachedParkThread::block_on::he6fb9cbd2e19c0d2
      at <unknown source file>:<unknown line>
   8: tokio::runtime::context::runtime::enter_runtime::h23806c7a69b68cf1
      at <unknown source file>:<unknown line>
   9: surrealdb_migrations::main::h6dc4d609382dc6ac
      at <unknown source file>:<unknown line>
  10: std::sys_common::backtrace::__rust_begin_short_backtrace::h3b5ad835699c4947
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::had3220220a969bf1
      at <unknown source file>:<unknown line>
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h82652f9ce79fee66
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:284
  13: std::panicking::try::do_call::he4b5337a350acea1
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504
  14: std::panicking::try::hd48750cb63f7f01f
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468
  15: std::panic::catch_unwind::h81e9ecf5c019a4bd
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142
  16: std::rt::lang_start_internal::{{closure}}::hef99b32d530f302f
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:148
  17: std::panicking::try::do_call::h223c3327abf6d3d9
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504
  18: std::panicking::try::ha6263cc242c2d045
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468
  19: std::panic::catch_unwind::h38c8a27284d90e98
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142
  20: std::rt::lang_start_internal::h2d2b3c117ba7283e
      at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:148
  21: main<unknown>
      at <unknown source file>:<unknown line>

[Odonno]

Hi @unidesigner

It looks like you are still using surrealdb beta9 (1.0.0-beta.9+20230402.5eafebd). Is it a miscopy? If not, can you try to update to the latest version of surrealdb and see if it changes anything.

In the same time, a 67MB migration is very huge. I may need to consider this edge case scenario and try to find a suitable solution when this is needed.

[unidesigner]

Hi @Odonno

Yes, it was a miscopy, I'm actually using 1.0.0+20230913.54aedcd for linux on x86_64

Hmm, I don't think it's that huge of a migration, it just inserts about 200k records. When I'm running it in a transaction with surreal import it imports within a couple of seconds (using http endpoint). So I don't see why this should not be working with surrealdb-migrations. Here's the migration file for testing. Thanks for looking into it!

[unidesigner]

Hi @Odonno

Do you think it will be straightforward to implement support for this edge case scenario? What are your thoughts on this issue?