Lightning-AI/litgpt

> From-scratch, single-file implementations of 20+ open LLMs with CLI recipes to pretrain, finetune, and deploy them.

GitHub repo · Official website · License: Apache-2.0

Overview

LitGPT is Lightning AI's library of open large language models, each reimplemented from scratch in a readable, single-file style with a deliberate policy of "no abstractions." It descends from the nanoGPT lineage by way of Lit-LLaMA, and grew into a catalog of 20+ model families (Llama 2/3, Gemma, Phi, Qwen, Mistral/Mixtral, Pythia, Falcon, OLMo, and others) sharing one config-driven training and inference stack[^1]. The pitch is that you can read the model code end to end without chasing base classes, while still getting production-grade features: Flash Attention, FSDP, tensor parallelism, LoRA/QLoRA/adapter finetuning, and fp4/8/16 quantization.

The intended audience splits two ways. For learners, the from-scratch code is documentation you can step through in a debugger. For practitioners, the litgpt CLI (download, finetune, pretrain, evaluate, serve, chat) is a batteries-included recipe runner that scales from one GPU to hundreds without rewriting the training loop. The defining tension is that these two audiences pull in opposite directions: the "no abstractions, one file per model" rule that makes the code teachable also means every new architecture is a hand-ported copy, and shared bugs or optimizations must be propagated across files rather than fixed in one base class.

It is not a general model hub or a serving platform. It is a curated set of weights-compatible reimplementations plus training recipes, tied by design to the PyTorch Lightning / Fabric ecosystem and, commercially, to Lightning AI's cloud[^2].

Getting Started

pip install 'litgpt[extra]'

from litgpt import LLM

llm = LLM.load("microsoft/phi-2")
text = llm.generate("Fix the spelling: Every fall, the family goes to the mountains.")
print(text)

# CLI: the same model families, driven by subcommands
litgpt download   list                              # list supported checkpoints
litgpt finetune   meta-llama/Llama-3.2-3B-Instruct  # LoRA/full finetune (auto-downloads weights)
litgpt serve      microsoft/phi-2                    # LitServe endpoint on :8000
litgpt evaluate   microsoft/phi-2 --tasks 'mmlu'     # via EleutherAI lm-eval-harness

Architecture / How It Works

At the center is a single generic transformer (litgpt/model.py) plus a Config dataclass. Rather than one class per architecture, LitGPT parameterizes a common GPT block over the axes that actually differ between modern decoder-only models: rotary embedding settings, normalization type (LayerNorm vs RMSNorm), MLP variant (GELU, LLaMA-style SwiGLU, GEMMA), attention grouping (MHA/MQA/GQA), MoE routing, and head/intermediate dimensions. Adding a model is mostly adding a Config entry with the right hyperparameters and a weight-conversion mapping, not writing a new forward pass. This is what "20+ from-scratch models" really means in practice: one flexible implementation, many named configurations.

Training and distribution run on Lightning Fabric, the lower-level sibling of the PyTorch Lightning Trainer. Fabric handles device placement, mixed precision, FSDP/DDP sharding, and TPU support without imposing the full Trainer loop, which keeps the training scripts explicit and hackable[^3]. Finetuning methods (litgpt/finetune/lora.py, adapter.py, adapter_v2.py, full.py) are separate scripts rather than flags on one mega-loop.

Weights are consumed by conversion: litgpt download pulls a HuggingFace checkpoint and convert_hf_checkpoint rewrites the tensors into LitGPT's naming. There is a reverse convert_lit_checkpoint for exporting back to HF format. This conversion layer is the seam where most "model X doesn't load" issues live, because it must track every upstream renaming of layers and rotary conventions. Configuration is YAML-first: the config_hub/ recipes pin hyperparameters, and the CLI is generated by jsonargparse, so nearly every dataclass field is a --flag.

Serving is delegated to LitServe, Lightning's inference server, exposed through litgpt serve. Evaluation is delegated to EleutherAI's lm-evaluation-harness. LitGPT owns the model code and recipes; it leans on sibling projects for the endpoints and benchmarks.

Production Notes

This is a training/finetuning toolkit, not a high-throughput inference server. litgpt serve runs the from-scratch PyTorch forward pass behind LitServe. It is fine for internal endpoints and demos, but it does not implement paged attention, continuous batching, or the throughput optimizations of dedicated serving stacks. For production inference on open weights, export to HF and serve with vLLM, TGI, or SGLang.
The conversion layer is the fragile boundary. Because models are reimplemented and weights are remapped, a newly released checkpoint (or an upstream config change) may not load until a Config and conversion entry are added. Pin the LitGPT version against the specific model you rely on; don't assume "any HF checkpoint of family X" works.
Gated weights need HuggingFace tokens. Llama, Gemma, and similar downloads require an accepted license and HF_TOKEN; litgpt download will fail without it. This is an operational step that surprises first-time users.
Memory footprint is configuration-driven, and defaults are not conservative. Full finetuning of larger models needs FSDP across multiple GPUs; single-GPU work generally means LoRA/QLoRA plus fp4/8 quantization. The recipes assume you match the YAML to your hardware. OOM during finetune usually means a mismatched recipe, not a bug.
API stability is limited. LitGPT is versioned as a fast-moving research library. Config names, CLI flags, and the Python LLM API have shifted across releases; recipes and checkpoint directories from an old version may not be forwarding-compatible. Treat pinned versions as part of your reproducibility contract.
The commercial gravity is real but non-coercive. The README funnels toward Lightning Cloud for GPUs, and Studios wrap the workflows, but the library itself is Apache-2.0 and runs fine on any PyTorch host, on-prem or otherwise[^2].

When to Use / When Not

Use when:

You want to read and modify an LLM's actual forward pass without an abstraction stack in the way.
You need reproducible pretrain/finetune recipes that scale from 1 to many GPUs/TPUs via Fabric.
You want LoRA/QLoRA/adapter finetuning of open weights with YAML-pinned configs.
You are teaching or learning transformer internals and want runnable, single-file references.

Avoid when:

You need maximum-throughput production inference — use a dedicated serving engine instead.
You want the widest possible model coverage the day each model drops — a hub like Transformers tracks releases faster.
You want a stable, rarely-changing API to build a long-lived product surface on.
You only need to call existing hosted models and never train or introspect them.

Alternatives

huggingface/transformers — use when you need the broadest model coverage and immediate support for new releases, at the cost of heavier abstraction.
karpathy/nanoGPT — use when you want the minimal single-file GPT for learning and don't need 20+ architectures or a CLI.
vllm-project/vllm — use when the goal is production inference throughput (paged attention, continuous batching), not training.
hiyouga/LLaMA-Factory — use when you want a broad finetuning UI/CLI over many models with less emphasis on from-scratch readable code.
unslothai/unsloth — use when single-GPU finetuning speed and memory efficiency are the priority.

History

| Version | Date | Notes | |---------|------|-------| | Lit-LLaMA | 2023-03 | Predecessor: from-scratch LLaMA on Fabric, seed of the "no abstractions" approach[^1]. | | Initial repo | 2023-05-04 | litgpt created, generalizing beyond a single model family. | | 0.1.x | 2024 | CLI consolidation (download/finetune/pretrain/evaluate), YAML config hub, LitData integration. | | 0.4–0.5 | 2024 | LLM high-level Python API; LitServe-backed serve; broadened model catalog. | | Ongoing | 2026 | 20+ families incl. Llama 3.x, Gemma 2/3, Phi 4, Qwen2.5/QwQ; Apache-2.0 throughout[^4]. |

References

[^1]: LitGPT README and model catalog. https://github.com/Lightning-AI/litgpt [^2]: Lightning AI (company and cloud behind LitGPT). https://lightning.ai [^3]: Lightning Fabric documentation. https://lightning.ai/docs/fabric/stable/ [^4]: Repository metadata (license Apache-2.0, created 2023-05-04) via GitHub API, retrieved 2026-07. https://github.com/Lightning-AI/litgpt

Wiki: Lightning-AI/litgpt