bigscience-workshop / petals Star 8.7k Code Issues Pull requests 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading nlp bloom distributed-systems machine-learning deep-learning chatbot pytorch falcon transformer neural-networks llama gpt pretrained-models language-models volunteer-computing pipeline-parallelism guanaco tensor-parallelism large-language-models llama2 Updated Apr 29, 2024 Python
kaiyuyue / torchshard Star 293 Code Issues Pull requests Slicing a PyTorch Tensor Into Parallel Shards pytorch model-parallelism tensor-parallelism Updated Jul 27, 2021 Python
xrsrke / pipegoose Star 73 Code Issues Pull requests Discussions Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)* transformers moe data-parallelism distributed-optimizers model-parallelism megatron mixture-of-experts pipeline-parallelism huggingface-transformers megatron-lm tensor-parallelism large-scale-language-modeling 3d-parallelism zero-1 sequence-parallelism Updated Dec 14, 2023 Python
aniquetahir / JORA Star 20 Code Issues Pull requests JORA: JAX Tensor-Parallel LoRA Library machine-learning lora jax tensor-parallelism large-language-models Updated Apr 25, 2024 Python
fattorib / transformer_shmap Star 10 Code Issues Pull requests Tensor Parallelism with JAX + Shard Map transformers gpt tpu jax tensor-parallelism pjit shmap Updated Sep 29, 2023 Python