{"full_name":"NVIDIA/TensorRT-LLM","owner":"NVIDIA","name":"TensorRT-LLM","shard":"1","url":"https://github.com/NVIDIA/TensorRT-LLM","description":"TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.","homepage":"https://nvidia.github.io/TensorRT-LLM","topics":["blackwell","cuda","moe","pytorch","llm-serving"],"tag_pages":["blackwell","cuda","llm-serving","moe","pytorch"],"language":"Python","license":"NOASSERTION","avatar":"https://avatars.githubusercontent.com/u/1728152?v=4","created_at":"2023-08-16T17:14:27Z","pushed_at":"2026-11-15T15:33:09Z","metrics":{"stars":15079,"forks":2936,"watchers":119,"issues":4252,"prs":17811,"releases":96,"issues_open":752,"issues_closed":3500,"prs_open":1340,"prs_closed":4368,"prs_merged":12103,"discussions":180},"growth":{"day":{"stars":4,"forks":1,"base_day":"2026-11-15"},"week":{"stars":77,"forks":34,"base_day":"2026-11-04"},"month":{"stars":277,"forks":124,"base_day":"2026-10-17"}},"history":{"dates":["2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[14756,14757,14761,14763,14767,14775,14777,14781,14787,14802,14808,14815,14822,14834,14843,14859,14878,14906,14920,14931,14937,14956,14962,14966,14974,14978,14992,15002,15051,15062,15069,15072,15073,15075,15079],"forks":[2792,2792,2795,2796,2797,2798,2800,2801,2806,2812,2815,2819,2822,2825,2831,2837,2842,2847,2851,2862,2862,2870,2872,2878,2881,2888,2898,2902,2924,2926,2929,2929,2932,2935,2936],"watchers":[122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,119,118,119,119,119,119,119,119,null,119,119,119,119,119,119],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,734,740,743,747,748,752],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1343,1346,1340,1336,1334,1340]},"boards":{},"same_owner":[{"full_name":"NVIDIA/DALI","name":"DALI","shard":"1","boards":{}},{"full_name":"NVIDIA/Isaac-GR00T","name":"Isaac-GR00T","shard":"1","boards":{}},{"full_name":"NVIDIA/Megatron-LM","name":"Megatron-LM","shard":"1","boards":{}},{"full_name":"NVIDIA/Model-Optimizer","name":"Model-Optimizer","shard":"1","boards":{}},{"full_name":"NVIDIA/NemoClaw","name":"NemoClaw","shard":"1","boards":{}},{"full_name":"NVIDIA/OpenShell","name":"OpenShell","shard":"1","boards":{"monthly":107,"monthly_rate":96}},{"full_name":"NVIDIA/SkillEvaluator","name":"SkillEvaluator","shard":"1","boards":{"daily":18,"weekly":138,"daily_rate":4,"weekly_rate":3,"monthly_rate":21,"daily_forks":18,"daily_watchers":189}},{"full_name":"NVIDIA/SkillSpector","name":"SkillSpector","shard":"1","boards":{"daily":148,"weekly":170,"monthly":91,"monthly_rate":110}},{"full_name":"NVIDIA/apex","name":"apex","shard":"1","boards":{}},{"full_name":"NVIDIA/cosmos","name":"cosmos","shard":"1","boards":{}},{"full_name":"NVIDIA/cuda-samples","name":"cuda-samples","shard":"1","boards":{}},{"full_name":"NVIDIA/cudf","name":"cudf","shard":"1","boards":{}},{"full_name":"NVIDIA/cudnn-frontend","name":"cudnn-frontend","shard":"1","boards":{}},{"full_name":"NVIDIA/cutlass","name":"cutlass","shard":"1","boards":{}},{"full_name":"NVIDIA/garak","name":"garak","shard":"1","boards":{}},{"full_name":"NVIDIA/nvidia-docker","name":"nvidia-docker","shard":"1","boards":{}},{"full_name":"NVIDIA/open-gpu-kernel-modules","name":"open-gpu-kernel-modules","shard":"1","boards":{}}],"issues":[{"number":2953,"title":"Running into free(): double free detected in tcache 2 when using trtllm-bench in a multi-node scenario","comments":32,"created_at":"2025-03-19T18:44:04Z"},{"number":5581,"title":"Running DeepSeek R1 on 8x RTX 6000 PRO","comments":24,"created_at":"2025-06-29T23:12:19Z"},{"number":3334,"title":"Support cpu inference with scaffolding","comments":21,"created_at":"2025-04-07T13:08:03Z"},{"number":2338,"title":"Whisper Encoder issues with Executor API","comments":18,"created_at":"2024-10-15T15:07:17Z"},{"number":12071,"title":"[Bug]: BeamSearch with beam_width 128 Encountered an error in cudaMemcpy2DAsync","comments":16,"created_at":"2026-03-10T06:32:53Z"},{"number":14500,"title":"`1.3.0rc15`: trtllm-serve hangs in fused_moe/quantization.py during cold-start on GB10 / sm_121 — `1.3.0rc12` works with identical config","comments":15,"created_at":"2026-05-24T00:54:18Z"},{"number":9705,"title":"[Usage]: How to perform Wide Expert Parallelism (Wide EP) inference testing for Qwen3-235B-A22B-FP8 on 8 GPUs using TensorRT LLM?","comments":13,"created_at":"2025-12-04T12:00:59Z"},{"number":3701,"title":"[Call for contributions] Feature parity between TensorRT and PyTorch backend - Quantization","comments":13,"created_at":"2025-04-20T06:38:34Z"}],"discussions":[{"number":16360,"title":"SM120 (RTX PRO 6000) MoE backend gaps: MoeConfig.backend schema missing MegaMoE entries, and no EP-capable path for this hardware","comments":0,"category":"General","created_at":"2026-07-14T05:43:10Z"},{"number":15508,"title":"Pre-inference context optimization: reducing prompt tokens before GPU","comments":0,"category":"Show and tell","created_at":"2026-06-20T21:25:26Z"},{"number":15476,"title":"CUDA cudaMemcpyAsync invalid argument error when using Triton instance_group multi-instance load balancing with TensorRT-LLM BLS model on RTX 5090","comments":2,"category":"Q&A","created_at":"2026-06-18T12:23:12Z"},{"number":14696,"title":"Hi Jhao-Ting","comments":0,"category":"General","created_at":"2026-05-28T14:28:04Z"},{"number":14425,"title":"Scaffolding idea","comments":1,"category":"Ideas","created_at":"2026-05-21T19:12:20Z"},{"number":12948,"title":"Is there context parallelism method using exact algorithm for MLA prefill in pytorch backend?","comments":0,"category":"General","created_at":"2026-04-11T08:17:44Z"},{"number":12308,"title":"Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether this hints at a more specialized future for language inference","comments":0,"category":"Show and tell","created_at":"2026-03-18T06:03:57Z"},{"number":12164,"title":"How can I do tool calling for gpt-oss-120B","comments":0,"category":"General","created_at":"2026-03-12T14:23:38Z"}],"latest_release":{"name":"v1.2.1","tag":"v1.2.1","published_at":"2026-04-20T11:51:33Z","body":"## Highlights\r\n\r\n- **Fixed Issue**\r\n  - Fixed an issue that caused KV cache corruption (#12770)\r\n\r\n- **Infrastructure Changes**\r\n  - Upgraded xgrammar and flashinfer (#12811)\r\n","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}