-
Notifications
You must be signed in to change notification settings - Fork 128
What would it take to train it for other languages? #16
Comments
Hey so basically the main things that would be needed are a) giant dataset of PHP code - the original codegen models are trained on the Github Bigquery Dataset and there is a subset for PHP so potentially that would be a good source b.i) a large enough machine to fine tune the model - I'm a hobbyist and in order to train a model with billions of parameters you need commercial GPUs (even on a 4090ti you would struggle to fine tune the 2B or larger models). OR b.ii) Use a low resource fine tuning pattern like LoRA which adapts the model by adding additional layers to the model that are specialised for the new language you want to target. This changes the shape/architecture of the model making it incompatible with the current implementation of codegen in ggml so I would need to make some changes to the cpp. llama.cpp recently added LoRa support so it is feasible that lora support could be added. |
I use PHP, could this be added?
The text was updated successfully, but these errors were encountered: