{"full_name":"huggingface/trl","owner":"huggingface","name":"trl","shard":"1","url":"https://github.com/huggingface/trl","description":"Train transformer language models with reinforcement learning.","homepage":"http://hf.co/docs/trl","topics":[],"tag_pages":[],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/25720743?v=4","created_at":"2020-03-27T10:54:55Z","pushed_at":"2026-11-15T05:37:43Z","metrics":{"stars":19741,"forks":3148,"watchers":104,"issues":2914,"prs":5283,"releases":104,"issues_open":164,"issues_closed":2750,"prs_open":137,"prs_closed":1154,"prs_merged":3992,"discussions":71},"growth":{"day":{"stars":6,"forks":3,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[19701,19721,19728,19729,19733,19735,19741],"forks":[3127,3134,3137,3138,3143,3145,3148],"watchers":[null,104,104,104,104,104,104],"issues_open":[null,166,161,161,192,163,164],"prs_open":[null,140,138,137,135,137,137]},"boards":{},"same_owner":[{"full_name":"huggingface/transformers","name":"transformers","shard":"1","boards":{"total":54}},{"full_name":"huggingface/OpenEnv","name":"OpenEnv","shard":"1","boards":{}},{"full_name":"huggingface/accelerate","name":"accelerate","shard":"1","boards":{}},{"full_name":"huggingface/agents-course","name":"agents-course","shard":"1","boards":{}},{"full_name":"huggingface/alignment-handbook","name":"alignment-handbook","shard":"1","boards":{}},{"full_name":"huggingface/diffusers","name":"diffusers","shard":"1","boards":{}},{"full_name":"huggingface/ml-intern","name":"ml-intern","shard":"1","boards":{}},{"full_name":"huggingface/parler-tts","name":"parler-tts","shard":"1","boards":{}},{"full_name":"huggingface/pytorch-image-models","name":"pytorch-image-models","shard":"1","boards":{}},{"full_name":"huggingface/sentence-transformers","name":"sentence-transformers","shard":"1","boards":{}},{"full_name":"huggingface/skills","name":"skills","shard":"1","boards":{}},{"full_name":"huggingface/smolagents","name":"smolagents","shard":"1","boards":{}},{"full_name":"huggingface/speech-to-speech","name":"speech-to-speech","shard":"1","boards":{}},{"full_name":"huggingface/transformers.js","name":"transformers.js","shard":"1","boards":{}}],"issues":[{"number":7202,"title":"Audit the trl-internal-testing repos that have no generation script","comments":14,"created_at":"2026-09-14T09:46:50Z"},{"number":3399,"title":"FP8 training","comments":14,"created_at":"2025-05-01T11:30:15Z"},{"number":3549,"title":"Packing sequences for memory efficiency in GRPO and other preference learning implementations","comments":12,"created_at":"2025-06-09T02:57:30Z"},{"number":7137,"title":"Align tiny test model configs with their reference models","comments":11,"created_at":"2026-09-09T15:06:00Z"},{"number":6857,"title":"Tiny test models are incompatible with Flash Attention (head_size=2)","comments":10,"created_at":"2026-08-21T14:30:53Z"},{"number":4611,"title":"SFTTrainer: Logging Experts Statistics","comments":9,"created_at":"2025-12-02T20:14:37Z"},{"number":3717,"title":"Add activation checkpointing to DPO and GRPO","comments":9,"created_at":"2025-07-10T04:11:07Z"},{"number":2580,"title":"GKDTrainer + FSDP results in RuntimeError: Expected all tensors to be on the same device, but found at least two devices","comments":9,"created_at":"2025-01-17T02:05:02Z"}],"discussions":[{"number":8257,"title":"loraspectra: Zero-dependency Rust CLI to audit spectral rank, prune noise singular values, and orthogonally merge SFT/DPO/GRPO LoRA .safetensors","comments":0,"category":"Show and tell","created_at":"2026-11-12T02:35:38Z"},{"number":7534,"title":"When should I choose SFT over GRPO for fine-tuning a small language model?","comments":1,"category":"Q&A","created_at":"2026-10-05T17:58:11Z"},{"number":7231,"title":"LoRA adapters trained on a 4-bit base degrade when served on a BF16 base — 100-sample comparison","comments":1,"category":"General","created_at":"2026-09-16T08:38:43Z"},{"number":7191,"title":"Taking on ongoing maintenance responsibilities","comments":0,"category":"General","created_at":"2026-09-12T07:22:41Z"},{"number":7082,"title":"You fine-tune on customer data with SFTTrainer. Can you prove it didn't memorize? memaudit runs that audit inside your training job.","comments":0,"category":"Show and tell","created_at":"2026-09-06T02:11:48Z"},{"number":5933,"title":"Proposal: Add TrOPD-style outlier FKL masking to GOLDTrainer","comments":1,"category":"Ideas","created_at":"2026-06-03T15:48:09Z"},{"number":5509,"title":"Decouple rollout from trainer process — RolloutService protocol","comments":1,"category":"Ideas","created_at":"2026-04-10T16:14:34Z"},{"number":5432,"title":"Is this a common/reasonable recipe for full finetuning Qwen3.5-4B?","comments":1,"category":"Q&A","created_at":"2026-04-01T22:01:35Z"}],"latest_release":{"name":"v1.18.0","tag":"v1.18.0","published_at":"2026-11-13T15:37:01Z","body":"<img width=\"1672\" height=\"941\" alt=\"TRL v1 18 motorcycle release\" src=\"https://github.com/user-attachments/assets/701191f4-46e1-45ea-95ad-60a18f19497a\" />\r\n\r\n\r\n## Features\r\n\r\n> [!WARNING]\r\n> The submodules of `trl.experimental.async_grpo` are gone. `from trl.experimental.async_grpo.async_grpo_config import AsyncGRPOConfig`, and imports from `async_rollout_worker`, `packing`, `vllm_client` or `weight_transfer` in that package, no longer work. Import the trainer from `trl`, the rollout loop from `trl.trainer.async_grpo_trainer`, and the vLLM client from `trl.generation.async_vllm_client` (renamed `AsyncVLLMClient`). `from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer` still works, with a `FutureWarning`, until v2.0.0.\r\n\r\n### :mortar_board: `AsyncGRPOTrainer` is now a stable trainer\r\n\r\n`AsyncGRPOTrainer` runs GRPO with generation decoupled from training. A rollout worker in a background process streams completions from a vLLM server, scores them and queues them, while the training loop pulls from the queue and updates the model. Generation and gradient updates overlap instead of alternating, and the trainer sends its new weights to the server over NCCL every `weight_sync_steps` steps.\r\n\r\nIn v1.17 it lived in `trl.experimental.async_grpo`: seven files, three ways to batch and bound generation, two constructor arguments for swapping out internals, and metric names that didn't match GRPO's. This cycle cleaned that up in about 40 PRs, tracked in https://github.com/huggingface/trl/issues/7841: config renames, one timeout instead of three, one batching mode, generation held to `max_staleness`, GRPO's metric names and init checks, a public rollout loop, and the two-file layout every stable trainer has. Then https://github.com/huggingface/trl/pull/8288 moved it to the top-level package.\r\n\r\n```python\r\n# Before\r\nfrom trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer\r\n\r\n# Now\r\nfrom trl import AsyncGRPOConfig, AsyncGRPOTrainer\r\n```\r\n\r\nThe old path still works and emits a `FutureWarning` (removal in v2.0.0). The rollout loop extension point lives next to the trainer, `from trl.trainer.async_grpo_trainer import AsyncRolloutLoop, Rollout`, and isn't exported from `trl`. The OpenEnv / Harbor harness stays experimental, as `HarnessAsyncGRPOTrainer` in `trl.experimental.async_grpo.openenv_harness`. Logged metric names can still change between releases.\r\n\r\nQuick start, from the [docs](https://huggingface.co/docs/trl/async_grpo_trainer). Install with `pip install trl[vllm,kernels,math_verify]`, then start the vLLM server on GPU 0:\r\n\r\n```bash\r\nCUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \\\r\n    --weight-transfer-config '{\"backend\": \"nccl\"}' \\\r\n    --logprobs-mode processed_logprobs \\\r\n    --api-server-count 8 \\\r\n    --max-model-len 4096\r\n```\r\n\r\nThen train on GPU 1 with `CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py`:\r\n\r\n```python\r\n# train_async_grpo.py\r\nfrom datasets import load_dataset\r\nfrom trl import AsyncGRPOTrainer\r\nfrom trl.rewards import accuracy_reward\r\n\r\nif __name__ == \"__main__\":  # the rollout worker is a spawned process that re-imports this script\r\n    dataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\r\n\r\n    trainer = AsyncGRPOTrainer(\r\n        model=\"Qwen/Qwen3.5-2B\",\r\n        reward_funcs=accuracy_reward,\r\n        train_dataset=dataset,\r\n    )\r\n    trainer.train()\r\n```\r\n\r\nThis exact example runs end to end. It's a sanity check, not a speedup comparison:\r\n\r\n| | |\r\n| --- | --- |\r\n| Model / data / reward | `Qwen/Qwen3.5-2B`, `trl-lib/DeepMath-103K`, `accuracy_reward` |\r\n| Hardware | 2x B200: vLLM on one GPU, the trainer on the other |\r\n| Config | default `AsyncGRPOConfig`: fp32 weights with bf16 autocast, 8 generations per prompt, up to 2048 completion tokens, `token_budget` read from `--max-model-len 4096` |\r\n| Step time | ~7 s/step, after ~5 min of warmup |\r\n| Throughput | ~7k trained tokens/s |\r\n| Trainer vs vLLM log-prob gap | 0.007 to 0.02 |\r\n\r\nWhat","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}