{"full_name":"xorbitsai/inference","owner":"xorbitsai","name":"inference","shard":"1","url":"https://github.com/xorbitsai/inference","description":"Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.","homepage":"https://inference.readthedocs.io","topics":["pytorch","llm","artificial-intelligence","machine-learning","whisper","inference","openai-api","gemma","llamacpp","vllm","qwen","diffusers","glm","glm-5-3","kimi"],"tag_pages":["artificial-intelligence","gemma","glm","inference","kimi","llamacpp","llm","machine-learning","openai-api","pytorch","qwen","vllm","whisper"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/109655068?v=4","created_at":"2023-06-14T07:05:04Z","pushed_at":"2026-11-15T16:16:56Z","metrics":{"stars":9657,"forks":901,"watchers":58,"issues":3012,"prs":3079,"releases":151,"issues_open":6,"issues_closed":3006,"prs_open":25,"prs_closed":255,"prs_merged":2799,"discussions":0},"growth":{"day":{"stars":2,"forks":1,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[9649,9653,9654,9655,9655,9655,9657],"forks":[898,898,898,897,897,900,901],"watchers":[null,58,58,58,58,58,58],"issues_open":[null,12,7,8,9,6,6],"prs_open":[null,39,28,27,28,28,25]},"boards":{},"same_owner":[],"issues":[{"number":4633,"title":"ENH: SDNQ Diffusers Quantizer","comments":30,"created_at":"2026-02-27T05:47:46Z"},{"number":6043,"title":"[Bug] v4.1.0: second replica of a vLLM model reports READY but hangs every routed request forever (multi-replica in one pod unusable)","comments":5,"created_at":"2026-11-09T03:12:10Z"},{"number":5797,"title":"[Docker CD] Failure on main","comments":3,"created_at":"2026-10-21T18:16:48Z"},{"number":5634,"title":"是否考虑增加\"提交+轮寻\"的异步接口模式","comments":2,"created_at":"2026-10-05T01:22:13Z"},{"number":6098,"title":"Ordinary serialized ModelActor streams keep producing after their consumer closes","comments":0,"created_at":"2026-11-13T05:13:44Z"},{"number":6083,"title":"[Feature] First-class support for System One decision models (Cloudflare Clef, pplx-decider) — exposing sglang's /v1/systemone","comments":0,"created_at":"2026-11-11T06:07:59Z"}],"discussions":[],"latest_release":{"name":"v4.2.0","tag":"v4.2.0","published_at":"2026-11-07T13:07:47Z","body":"# What's new in 4.2.0 (2026-11-07)\n\n## Highlights\n\n- **Xavier prefix cache sharing.** Reuse compatible prefixes across replicas running vLLM, SGLang, TRT-LLM, or MLX. Xavier expands LMCache storage support with Mooncake Store and can prioritize retaining KV prefixes while agents wait for tool results, reducing repeated prefill when they resume. ([#5917](https://github.com/xorbitsai/inference/pull/5917), [#5905](https://github.com/xorbitsai/inference/pull/5905), [#6018](https://github.com/xorbitsai/inference/pull/6018))\n\n- **Software updates and rolling model reloads.** Administrators can check for and install Xinference updates in the Web UI, upgrading Workers, the Supervisor, and the API in order. Supported vLLM and SGLang deployments can retain GPU weights and reload one replica at a time while in-flight requests finish on the previous engine instance. ([#5873](https://github.com/xorbitsai/inference/pull/5873), [#5944](https://github.com/xorbitsai/inference/pull/5944))\n\n- **Start model benchmarks from the Web UI.** Run tests at different concurrency levels against a running chat model, compare throughput, time to first token (TTFT), and time per output token (TPOT) across runs, and export results as JSON. ([#5904](https://github.com/xorbitsai/inference/pull/5904))\n\n- **Native reasoning controls and request priority across LLM engines.** The OpenAI- and Anthropic-compatible APIs expose native reasoning effort for supported models. The OpenAI-compatible API also supports a thinking budget on compatible runtimes, and the Web UI shows controls when available. Chat and text completion requests use a common `priority` parameter across all supported LLM engines, subject to engine version and configuration. ([#6001](https://github.com/xorbitsai/inference/pull/6001), [#6019](https://github.com/xorbitsai/inference/pull/6019))\n\n- **Real-time Qwen3 audio.** Serve concurrent real-time speech recognition with Qwen3-ASR and streaming speech synthesis with Qwen3-TTS CustomVoice through vLLM. ([#5993](https://github.com/xorbitsai/inference/pull/5993))\n\nThese are the changes in inference v4.2.0.\n## New features\n* feat: support service relaunch on Windows by @qinxuye in https://github.com/xorbitsai/inference/pull/5868\n* feat: check and install Xinference updates from the Web UI by @qinxuye in https://github.com/xorbitsai/inference/pull/5873\n* FEAT: Add optional Worker controller-session admission by @qinxuye in https://github.com/xorbitsai/inference/pull/5901\n* feat: add Mooncake Store storage to Xavier LMCache by @qinxuye in https://github.com/xorbitsai/inference/pull/5905\n* feat: support llama.cpp engine for DeepSeek-V4.1-Flash-1017 by @llyycchhee in https://github.com/xorbitsai/inference/pull/5912\n* feat(ocr): add Unlimited-OCR-2 support by @Minamiyama in https://github.com/xorbitsai/inference/pull/5913\n* feat(ocr): add OvisOCR3 support by @Minamiyama in https://github.com/xorbitsai/inference/pull/5914\n* feat(ocr): add MiniCPM-OCR-1.0 support by @Minamiyama in https://github.com/xorbitsai/inference/pull/5916\n* feat: add Qwen NVFP4/YaRN support and update Gemma-4 GGUF runtime by @qinxuye in https://github.com/xorbitsai/inference/pull/5919\n* FEAT: Add optional Supervisor control extension mechanisms by @qinxuye in https://github.com/xorbitsai/inference/pull/5911\n* FEAT: Share Xavier prefix caches across engines by @qinxuye in https://github.com/xorbitsai/inference/pull/5917\n* feat: automatically retain GPU weights on supported model launches by @qinxuye in https://github.com/xorbitsai/inference/pull/5924\n* FEAT: Extend Qwen automatic YaRN engine support by @qinxuye in https://github.com/xorbitsai/inference/pull/5925\n* feat(video): add MiniMax-H3 to vLLM and SGLang with Lightning by @qinxuye in https://github.com/xorbitsai/inference/pull/5930\n* FEAT: Add pluggable control calls and Supervisor journal storage by @qinxuye in https://github.com/xorbitsai/inference/pull/5931\n* feat: support multi-replica weight retention and rolling reload by @qin","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}