Litgpt

LitGPT is an open-source tool for scalable pretraining, finetuning, and deployment of large language models with multi-GPU/TPU support.