Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Import should stream data rather than reading the whole file into memory #2189

Open
rathboma opened this issue May 8, 2024 · 0 comments
Open

BUG: Import should stream data rather than reading the whole file into memory #2189

rathboma opened this issue May 8, 2024 · 0 comments
Labels
accepted 👍 bug Something isn't working

Comments

@rathboma
Copy link
Collaborator

rathboma commented May 8, 2024

Large files break the import process for two reasons:

  1. The app runs out of memory
  2. We send 'too large' query text to the database driver, which causes an error

What we should do is:

  1. start a transactions
  2. Read the file record by record
  3. batch into ~x inserts / updates at once
  4. send those to the db in batches
  5. Close the file and transaction
@rathboma rathboma added accepted 👍 bug Something isn't working labels May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accepted 👍 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rathboma