{"full_name":"pytorch/torchtitan","owner":"pytorch","name":"torchtitan","shard":"1","url":"https://github.com/pytorch/torchtitan","description":"A PyTorch native platform for training generative AI models","homepage":"","topics":[],"tag_pages":[],"language":"Python","license":"BSD-3-Clause","avatar":"https://avatars.githubusercontent.com/u/21003710?v=4","created_at":"2023-12-13T01:51:37Z","pushed_at":"2026-11-15T11:34:52Z","metrics":{"stars":5909,"forks":1080,"watchers":58,"issues":896,"prs":5018,"releases":6,"issues_open":112,"issues_closed":784,"prs_open":327,"prs_closed":1717,"prs_merged":2974,"discussions":24},"growth":{"day":{"stars":3,"forks":-1,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[5889,5893,5898,5898,5903,5906,5909],"forks":[1077,1078,1080,1080,1081,1081,1080],"watchers":[null,58,58,58,58,58,58],"issues_open":[null,90,90,106,112,110,112],"prs_open":[null,338,341,344,331,327,327]},"boards":{},"same_owner":[{"full_name":"pytorch/pytorch","name":"pytorch","shard":"1","boards":{"total":132}},{"full_name":"pytorch/tutorials","name":"tutorials","shard":"1","boards":{}},{"full_name":"pytorch/vision","name":"vision","shard":"1","boards":{}}],"issues":[{"number":4272,"title":"Kimi K3 issue tracking","comments":20,"created_at":"2026-08-21T20:01:52Z"},{"number":5019,"title":"NCCL all-reduce OOM despite reclaimable PyTorch cache during Kimi K3 training","comments":15,"created_at":"2026-10-02T03:35:24Z"},{"number":431,"title":"Question about Pipeline parallelism","comments":13,"created_at":"2024-06-27T15:31:52Z"},{"number":3336,"title":"Investigate mismatch between cpu received tokens and cuda received tokens in EP","comments":11,"created_at":"2026-05-13T02:08:23Z"},{"number":3368,"title":"[question] Why is the load balancing disabled for multi-mask models","comments":10,"created_at":"2026-05-15T08:51:50Z"},{"number":3123,"title":"[SimpleFSDP] SimpleFSDP silently ignores `enable_fsdp_float8_all_gather=True`","comments":8,"created_at":"2026-04-28T00:22:50Z"},{"number":5692,"title":"[RFC] Support additional training diagnostics in TorchTitan","comments":7,"created_at":"2026-10-29T10:32:22Z"},{"number":4349,"title":"[RFC] Support Anticipatory Routing to mitigate MoE training loss spikes","comments":6,"created_at":"2026-08-27T13:15:14Z"}],"discussions":[{"number":3911,"title":"Validation Loss Plateaus only after 2000 steps Pretraining Regardless of LR Schedule (Follow-up to #3868)","comments":3,"category":"Q&A","created_at":"2026-07-13T19:17:33Z"},{"number":3882,"title":"Can I use 'torchtitan' for any model architecture?","comments":1,"category":"Q&A","created_at":"2026-07-07T21:22:12Z"},{"number":3868,"title":"How should I balance learning rate and data sampling during CPT on multiple datasets?","comments":6,"category":"Q&A","created_at":"2026-07-06T18:56:08Z"},{"number":3589,"title":"Requesting some insight on time spent during recovery when training with torchFT enabled","comments":1,"category":"Q&A","created_at":"2026-06-09T15:20:44Z"},{"number":2918,"title":"Usage of nn.Identity","comments":1,"category":"General","created_at":"2026-04-09T18:47:58Z"},{"number":2893,"title":"MFU calculation for Quantized Workloads","comments":1,"category":"General","created_at":"2026-04-08T20:43:09Z"},{"number":2796,"title":"torchtitan is featured on Shypd — 84K+ AI tools","comments":0,"category":"General","created_at":"2026-04-02T21:34:35Z"},{"number":2790,"title":"RoPE config: enforcing `dim`","comments":1,"category":"General","created_at":"2026-04-02T12:32:32Z"}],"latest_release":{"name":"v0.4.0","tag":"v0.4.0","published_at":"2026-10-28T18:05:38Z","body":"# TorchTitan 0.4.0\r\n\r\nTorchTitan 0.4.0 aligns with PyTorch 2.15, expands model coverage, improves large-scale MoE training, and unifies core training and TitanRL on a shared engine.\r\n\r\nThese highlights cover changes since 0.3.0. For broader updates and benchmarks, see [TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL](https://pytorch.org/blog/torchtitan-update-new-sota-models-performance-at-scale-titanrl/).\r\n\r\n## New model implementations\r\n\r\n- **Muse Spark Open:** Adds the 816B-A42B model, including text and vision/audio configurations. Multimodal CUDA graphs remain unsupported. ([#5420](https://github.com/pytorch/torchtitan/pull/5420))\r\n- **Kimi K3:** Hybrid KDA/MLA attention, LatentMoE, and vision support, with tensor/sequence and expert parallelism plus a dedicated pipeline implementation for attention residuals. ([#4025](https://github.com/pytorch/torchtitan/pull/4025), [#4312](https://github.com/pytorch/torchtitan/pull/4312))\r\n- **DeepSeek V4:** Initial Flash/Pro support with sparse attention, indexer loss, optional multi-token prediction, and debug recipes. Full-scale checkpoint and convergence validation remain outstanding. ([#3634](https://github.com/pytorch/torchtitan/pull/3634), [#4835](https://github.com/pytorch/torchtitan/pull/4835))\r\n\r\n## Training performance and memory\r\n\r\n- **Dist-MoE integration:** Fuses expert dispatch, computation, and combine. DeepSeek V3 BF16/MXFP8 recipes for NVIDIA SM100+ GPUs include fused MLA and SwiGLU. ([#4541](https://github.com/pytorch/torchtitan/pull/4541), [#4542](https://github.com/pytorch/torchtitan/pull/4542))\r\n- **Low-precision training:** MXFP8 gains FSDP-managed prepared weights and fused weight-gradient accumulation across microbatches. Experimental NVFP4 support extends to grouped MoE linear layers. Both paths require supported Blackwell hardware and their respective optional dependencies. ([#4203](https://github.com/pytorch/torchtitan/pull/4203), [#4911](https://github.com/pytorch/torchtitan/pull/4911), [#4821](https://github.com/pytorch/torchtitan/pull/4821))\r\n- **Local compilation:** `model.local_compile_regions` selects functions such as SwiGLU, RoPE, and gated RMSNorm for independent compilation and kernel fusion with `torch.compile`. ([#4894](https://github.com/pytorch/torchtitan/pull/4894), [#4931](https://github.com/pytorch/torchtitan/pull/4931), [#4985](https://github.com/pytorch/torchtitan/pull/4985), [#5026](https://github.com/pytorch/torchtitan/pull/5026))\r\n- **Faster router and LM-head layers:** `HiMidLoLinear` uses BF16 matrix multiplications with FP32 outputs. Its backward pass combines two or three BF16 parts of each FP32 output gradient, improving accuracy over a single BF16 cast. ([#4923](https://github.com/pytorch/torchtitan/pull/4923))\r\n- **CUDA graphs and pipeline memory:** Adds pipeline forward/backward capture, shared training/validation graph memory, and per-microbatch-group replay, plus controls for FSDP parameter residency and unshard lookahead. ([#4493](https://github.com/pytorch/torchtitan/pull/4493), [#5241](https://github.com/pytorch/torchtitan/pull/5241), [#4592](https://github.com/pytorch/torchtitan/pull/4592))\r\n- **Context parallelism:** Adds Ulysses support for standard and variable-length attention, improves MLA communication, and adds native context-parallel paths for Qwen3.5 Gated DeltaNet and Kimi K3 KDA. ([#4450](https://github.com/pytorch/torchtitan/pull/4450), [#4760](https://github.com/pytorch/torchtitan/pull/4760), [#5184](https://github.com/pytorch/torchtitan/pull/5184))\r\n- **Activation rematerialization:** `torch_remat` controls activation saving and recomputation by region. DeepEP dispatch preserves token order during recomputation, preventing incorrect MoE gradients. ([#4497](https://github.com/pytorch/torchtitan/pull/4497), [#5115](https://github.com/pytorch/torchtitan/pull/5115))\r\n- **FlexShard / DistMuon:** Extends optimizer-compute layouts with variable-sized `BlockShard` partitions and BF16 redistribution ","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}