Request to add 4-bit AdamW

[LiutongZhou]

Paper and Code

Paper: Memory Efficient Optimizers with 4-bit States
Code : https://github.com/thu-ml/low-bit-optimizers/blob/main/lpmm/optim/optimizer.py

[kozistr]

thanks for the request! it requires cuda files to compile like bitsandbytes optimizer, so it's complicated to move entire implementations into this project, instead, maybe we can support the optimizer like this example, that manually install or compile, and use it through pytorch-optimizer.

I'll figure it out how to smoothly support this in pytorch-optimizer :)

thank you!

[LiutongZhou]

Liked the idea of supporting it like the example.

Maybe even easier, make the FR to bnb and include it here?

Made the FR to TimDettmers/bitsandbytes#775

[kozistr]

Liked the idea of supporting it like the example.

Maybe even easier, make the FR to bnb and include it here?

Made the FR to TimDettmers/bitsandbytes#775

that's a great idea! it'd be much better that bnb supports 4-bit optimizers. then, for now, wait for the response and decide whether including both or only bnb.