{"full_name":"NVIDIA/Megatron-LM","owner":"NVIDIA","name":"Megatron-LM","shard":"1","url":"https://github.com/NVIDIA/Megatron-LM","description":"Ongoing research training transformer models at scale","homepage":"https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/quickstart.html","topics":["large-language-models","model-para","transformers"],"tag_pages":["large-language-models","transformers"],"language":"Python","license":"NOASSERTION","avatar":"https://avatars.githubusercontent.com/u/1728152?v=4","created_at":"2019-03-21T16:15:52Z","pushed_at":"2026-11-15T19:18:22Z","metrics":{"stars":18406,"forks":4735,"watchers":162,"issues":1992,"prs":7276,"releases":55,"issues_open":508,"issues_closed":1484,"prs_open":1570,"prs_closed":2069,"prs_merged":3637,"discussions":135},"growth":{"day":{"stars":12,"forks":1,"base_day":"2026-11-15"},"week":{"stars":85,"forks":35,"base_day":"2026-11-04"},"month":{"stars":283,"forks":120,"base_day":"2026-10-17"}},"history":{"dates":["2026-09-07","2026-09-08","2026-09-09","2026-09-10","2026-09-11","2026-09-12","2026-09-13","2026-09-14","2026-09-15","2026-09-16","2026-09-17","2026-09-18","2026-09-19","2026-09-20","2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[17589,17615,17631,17648,17655,17667,17701,17711,17734,17749,17750,17776,17787,17815,17840,17862,17866,17895,17913,17920,17932,17940,17948,17967,17979,17991,18003,18008,18012,18027,18031,18050,18055,18064,18075,18078,18080,18087,18099,18110,18123,18138,18152,18163,18169,18178,18197,18215,18230,18241,18253,18259,18274,18280,18289,18294,18299,18317,18321,18365,18377,18383,18385,18390,18394,18406],"forks":[4406,4416,4423,4424,4426,4436,4446,4451,4454,4456,4456,4464,4466,4479,4488,4494,4495,4504,4512,4520,4526,4530,4536,4545,4548,4557,4563,4566,4569,4574,4581,4587,4586,4587,4588,4590,4593,4598,4603,4607,4615,4621,4627,4628,4631,4633,4637,4645,4648,4655,4661,4665,4671,4676,4681,4684,4689,4697,4700,4723,4726,4725,4725,4730,4734,4735],"watchers":[167,167,167,168,168,169,172,172,172,172,172,173,173,173,173,173,172,172,172,172,172,172,172,172,172,172,172,171,171,171,171,171,171,171,171,171,171,171,171,171,171,170,170,170,170,170,170,171,171,172,172,167,167,167,167,167,167,166,166,null,162,162,162,162,162,162],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,503,508,507,509,508,508],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1533,1547,1550,1563,1561,1570]},"boards":{},"same_owner":[{"full_name":"NVIDIA/DALI","name":"DALI","shard":"1","boards":{}},{"full_name":"NVIDIA/Isaac-GR00T","name":"Isaac-GR00T","shard":"1","boards":{}},{"full_name":"NVIDIA/Model-Optimizer","name":"Model-Optimizer","shard":"1","boards":{}},{"full_name":"NVIDIA/NemoClaw","name":"NemoClaw","shard":"1","boards":{}},{"full_name":"NVIDIA/OpenShell","name":"OpenShell","shard":"1","boards":{"monthly":107,"monthly_rate":96}},{"full_name":"NVIDIA/SkillEvaluator","name":"SkillEvaluator","shard":"1","boards":{"daily":18,"weekly":138,"daily_rate":4,"weekly_rate":3,"monthly_rate":21,"daily_forks":18,"daily_watchers":189}},{"full_name":"NVIDIA/SkillSpector","name":"SkillSpector","shard":"1","boards":{"daily":148,"weekly":170,"monthly":91,"monthly_rate":110}},{"full_name":"NVIDIA/TensorRT-LLM","name":"TensorRT-LLM","shard":"1","boards":{}},{"full_name":"NVIDIA/apex","name":"apex","shard":"1","boards":{}},{"full_name":"NVIDIA/cosmos","name":"cosmos","shard":"1","boards":{}},{"full_name":"NVIDIA/cuda-samples","name":"cuda-samples","shard":"1","boards":{}},{"full_name":"NVIDIA/cudf","name":"cudf","shard":"1","boards":{}},{"full_name":"NVIDIA/cudnn-frontend","name":"cudnn-frontend","shard":"1","boards":{}},{"full_name":"NVIDIA/cutlass","name":"cutlass","shard":"1","boards":{}},{"full_name":"NVIDIA/garak","name":"garak","shard":"1","boards":{}},{"full_name":"NVIDIA/nvidia-docker","name":"nvidia-docker","shard":"1","boards":{}},{"full_name":"NVIDIA/open-gpu-kernel-modules","name":"open-gpu-kernel-modules","shard":"1","boards":{}}],"issues":[{"number":5660,"title":"[MCore][TE][NVFP4] Persist activation global input_scale/input_amax for offline FP4 inference","comments":26,"created_at":"2026-07-06T02:00:11Z"},{"number":3945,"title":"[BUG] \"Spike no more\" feature was removed","comments":18,"created_at":"2026-03-19T12:10:01Z"},{"number":7452,"title":"Silent numerical-correctness bugs in Megatron Core parallelism composition","comments":16,"created_at":"2026-09-17T21:16:34Z"},{"number":6714,"title":"MFSDP v2: overlap DP-inner and DP-outer communication","comments":16,"created_at":"2026-08-20T19:22:47Z"},{"number":4088,"title":"Hyperparameter Transfer beyond MuP","comments":16,"created_at":"2026-04-01T04:13:09Z"},{"number":4091,"title":"Add MLA support for Muon optimizer","comments":15,"created_at":"2026-04-01T08:37:52Z"},{"number":1445,"title":"[BUG]loss error when using MLA","comments":15,"created_at":"2025-03-03T10:20:39Z"},{"number":2797,"title":"Feature Request: Dual Chunk Attention for Long Context","comments":15,"created_at":"2026-01-02T16:58:22Z"}],"discussions":[{"number":7497,"title":"Scaling Bitwise-Deterministic Pretraining for a Trillion-Parameter Nemotron Model with NVIDIA Megatron Core","comments":0,"category":"Announcements","created_at":"2026-09-18T22:43:32Z"},{"number":5002,"title":"What If Megatron-LM Had Actual Physics?","comments":0,"category":"Ideas","created_at":"2026-05-26T22:58:04Z"},{"number":4061,"title":"Megatron LM is featured on Shypd — 80K+ AI tools","comments":0,"category":"General","created_at":"2026-03-30T17:54:36Z"},{"number":3915,"title":"Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether some language capability may eventually move outside the main dense training path","comments":0,"category":"Show and tell","created_at":"2026-03-18T07:29:41Z"},{"number":2438,"title":"Why doesn't theoretical memory take into account the expert parallelism and the partitioning of experts?","comments":0,"category":"General","created_at":"2025-12-01T07:51:04Z"},{"number":1640,"title":"How to set hierarchical CP group sizes to enable cp_comm_type=\"a2a+p2p\"","comments":1,"category":"Q&A","created_at":"2025-06-20T01:56:27Z"},{"number":1577,"title":"[QUESTION]About HybridDeviceOptimizer CPU Offloading Behavior","comments":1,"category":"Q&A","created_at":"2025-05-14T09:55:15Z"},{"number":1558,"title":"About the issue of interleaved pipeline scheduling","comments":0,"category":"Q&A","created_at":"2025-04-29T14:34:57Z"}],"latest_release":{"name":"NVIDIA Megatron Core 0.20.0","tag":"core_v0.20.0","published_at":"2026-10-29T21:06:49Z","body":"#### MoE\r\n\r\n* **DeepEP v2 dispatcher:** adds an experimental DeepEP v2 backend for the flex token dispatcher ([\\#5153](https://github.com/NVIDIA/Megatron-LM/pull/5153)).  \r\n* **NCCL EP MXFP8 communication (experimental):** adds MXFP8 dispatch-forward and combine-backward communication to the NCCL EP backend ([\\#6524](https://github.com/NVIDIA/Megatron-LM/pull/6524)).  \r\n* **NCCL EP eager mode**: now NCCL EP supports both eager mode and full-iteration graph mode ([\\#6229](https://github.com/NVIDIA/Megatron-LM/pull/6229)).  \r\n* [**ShortcutMoE**](https://arxiv.org/pdf/2404.05019)**:** adds shortcut-connected routing in HybridStack and overlaps expert all-to-all communication with paired attention or Mamba computation; CUDA graphs are not supported ([\\#6959](https://github.com/NVIDIA/Megatron-LM/pull/6959)).\r\n\r\n#### Model Architecture and Support\r\n\r\n* **DeepSeek V4:** fully supported by the next-gen `HybridModel` class ([\\#6403](https://github.com/NVIDIA/Megatron-LM/pull/6403), [\\#6404](https://github.com/NVIDIA/Megatron-LM/pull/6404), [\\#6405](https://github.com/NVIDIA/Megatron-LM/pull/6405), [\\#7469](https://github.com/NVIDIA/Megatron-LM/pull/7469)).  \r\n* **Manifold-constrained hyper-connections (mHC):** adds fused mHC connections to HybridModel, including pipeline and virtual-pipeline parallel support ([\\#6401](https://github.com/NVIDIA/Megatron-LM/pull/6401), [\\#7336](https://github.com/NVIDIA/Megatron-LM/pull/7336)).  \r\n* **Gated Delta Product (GDP):** adds GDP layers and model-building support for hybrid architectures ([\\#6074](https://github.com/NVIDIA/Megatron-LM/pull/6074)).  \r\n* **Wide residual streams:** adds streamwise residual connections, selective residual-stream recomputation, and integration with MTP and MIMO; currently requires PP=1 and CUDA graphs disabled ([\\#6716](https://github.com/NVIDIA/Megatron-LM/pull/6716), [\\#6805](https://github.com/NVIDIA/Megatron-LM/pull/6805), [\\#6855](https://github.com/NVIDIA/Megatron-LM/pull/6855), [\\#7440](https://github.com/NVIDIA/Megatron-LM/pull/7440)).  \r\n* **MTP-only training:** adds a mode that freezes the base model and trains only Multi-Token Prediction heads ([\\#7021](https://github.com/NVIDIA/Megatron-LM/pull/7021)).\r\n\r\n#### Parallelism\r\n\r\n* **Megatron-FSDP v2 integration:** adds the experimental Megatron Core adapter and Megatron Bridge, with context, expert, pipeline, and hybrid data-parallel sharding support; the adapter requires TP=1 and expert TP=1 ([\\#5865](https://github.com/NVIDIA/Megatron-LM/pull/5865), [\\#6450](https://github.com/NVIDIA/Megatron-LM/pull/6450), [\\#6486](https://github.com/NVIDIA/Megatron-LM/pull/6486), [\\#7220](https://github.com/NVIDIA/Megatron-LM/pull/7220), [\\#6873](https://github.com/NVIDIA/Megatron-LM/pull/6873)).  \r\n* **Megatron-FSDP v2 checkpointing:** adds PyTorch distributed checkpoint save/load for model and optimizer state; optimizer checkpointing currently excludes expert-parallel layouts ([\\#6024](https://github.com/NVIDIA/Megatron-LM/pull/6024), [\\#6215](https://github.com/NVIDIA/Megatron-LM/pull/6215)).  \r\n* Megatron-FSDP v2 supports hybrid FSDP.   \r\n  * The user can specify different ZeRO levels for inter-nvlink domains and intra-nvlink domains.   \r\n  * The user can specify different configs for the dense section and the MoE section.   \r\n* **Generalized Tensor Parallelism (GTP):** adds configurable batch/sequence partitioning, opt-in latent-projection sharding, and MTP support ([\\#7748](https://github.com/NVIDIA/Megatron-LM/pull/7748), [\\#6383](https://github.com/NVIDIA/Megatron-LM/pull/6383), [\\#6242](https://github.com/NVIDIA/Megatron-LM/pull/6242)).\r\n\r\n#### Performance and Memory\r\n\r\n* **Determinism:** multi-release effort, please read [our blog post](https://developer.nvidia.com/blog/scale-bitwise-deterministic-pretraining-with-nvidia-megatron-core/) to learn more  \r\n* **Megatron-FSDP v2 CUDA graphs:** adds full-iteration graph capture and replay for supported dense-model configurations ([\\#7075](https://github.com/NVIDIA/Me","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}