Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk Create without returning objects #1614

Open
Abdeldjalil-H opened this issue May 14, 2024 · 3 comments
Open

Bulk Create without returning objects #1614

Abdeldjalil-H opened this issue May 14, 2024 · 3 comments

Comments

@Abdeldjalil-H
Copy link
Contributor

The main problem here is that bulk_create converts objects to a list before insertion which loads all the models at the same time. The only reason for this is to return the inserted objects after the query is done. However, those objects are not being updated (adding pk for example), i.e the mothod will return what you sent. I think we can enhance this and make bulk_create benefits from #1613 .

Describe the solution you'd like
There are several approaches for this:

  1. Do not return anything from bulk_create or return the number of inserted rows. This maybe a breaking change.
  2. Add a keyword argument called return_objects or something like that to control the behaviour.
  3. Do not return anything when batch_size is set.

If none of the above seems logical, I think at least we should mention what is going behind the scenes in the docs, so the user can handle chunking on his own. And maybe use tuple(objects) instead of list(objects).

Describe alternatives you've considered
The most natural alternative is to chunk data on your side and leave batch_size as None. But this makes batch_size useless.

@abondar
Copy link
Member

abondar commented May 14, 2024

Hi!

I think ideally would be to support id population for backends that allow it to do in same query - as with RETURNING in postgres
But not sure I will be able to do it in any near time

As right now returning objects doesn't bring any benefits, as we return what we got in - I don't see much value in this return and we can change it

I think most explicit way to do that - would be for now just removing that return object, allowing to iterate through models without loading them all in memory

Next step could be adding return_objects param and allowing it only for supported databases, where we would return updated objects from db in same query

@Abdeldjalil-H
Copy link
Contributor Author

Hi @abondar . I can make a PR on that, removing the returned object.

@abondar
Copy link
Member

abondar commented May 14, 2024

If you could - I would gladly help you with reviewing and merging it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants