{"full_name":"alibaba/MNN","owner":"alibaba","name":"MNN","shard":"1","url":"https://github.com/alibaba/MNN","description":"MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.","homepage":"","topics":["mnn","convolution","vulkan","embedded-devices","winograd-algorithm","machine-learning","deep-learning","ml","arm","llm","transformer"],"tag_pages":["arm","deep-learning","llm","machine-learning","ml","transformer","vulkan"],"language":"C++","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/1961952?v=4","created_at":"2019-04-15T07:40:18Z","pushed_at":"2026-11-13T13:09:33Z","metrics":{"stars":16534,"forks":2534,"watchers":237,"issues":3636,"prs":1362,"releases":64,"issues_open":39,"issues_closed":3597,"prs_open":49,"prs_closed":304,"prs_merged":1009,"discussions":13},"growth":{"day":{"stars":7,"forks":2,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[16470,16495,16506,16505,16521,16527,16534],"forks":[2516,2520,2524,2524,2530,2532,2534],"watchers":[null,237,237,237,238,237,237],"issues_open":[null,42,43,43,43,38,39],"prs_open":[null,31,35,36,37,44,49]},"boards":{},"same_owner":[{"full_name":"alibaba/AliSQL","name":"AliSQL","shard":"1","boards":{}},{"full_name":"alibaba/Chat2DB","name":"Chat2DB","shard":"1","boards":{}},{"full_name":"alibaba/DataX","name":"DataX","shard":"1","boards":{}},{"full_name":"alibaba/QLExpress","name":"QLExpress","shard":"1","boards":{}},{"full_name":"alibaba/Sentinel","name":"Sentinel","shard":"1","boards":{}},{"full_name":"alibaba/arthas","name":"arthas","shard":"1","boards":{}},{"full_name":"alibaba/canal","name":"canal","shard":"1","boards":{}},{"full_name":"alibaba/druid","name":"druid","shard":"1","boards":{}},{"full_name":"alibaba/easyexcel","name":"easyexcel","shard":"1","boards":{}},{"full_name":"alibaba/flutter-go","name":"flutter-go","shard":"1","boards":{}},{"full_name":"alibaba/formily","name":"formily","shard":"1","boards":{}},{"full_name":"alibaba/higress","name":"higress","shard":"1","boards":{}},{"full_name":"alibaba/ice","name":"ice","shard":"1","boards":{}},{"full_name":"alibaba/jetcache","name":"jetcache","shard":"1","boards":{}},{"full_name":"alibaba/jvm-sandbox","name":"jvm-sandbox","shard":"1","boards":{}},{"full_name":"alibaba/lowcode-engine","name":"lowcode-engine","shard":"1","boards":{}},{"full_name":"alibaba/nacos","name":"nacos","shard":"1","boards":{}},{"full_name":"alibaba/open-code-review","name":"open-code-review","shard":"1","boards":{"daily":14,"weekly":49,"monthly":98,"daily_rate":107,"weekly_rate":197,"daily_forks":5,"daily_watchers":139}},{"full_name":"alibaba/p3c","name":"p3c","shard":"1","boards":{}},{"full_name":"alibaba/page-agent","name":"page-agent","shard":"1","boards":{}}],"issues":[{"number":4802,"title":"Qwen2.5-Omni-3B/7B 在最新代码上opencl上推理崩溃","comments":14,"created_at":"2026-08-24T05:46:00Z"},{"number":4883,"title":"qwen3.5的mnn模型如何进行gpu推理","comments":12,"created_at":"2026-09-17T09:17:36Z"},{"number":4641,"title":"［bug］Gemma-4-E2B-it-MNN在识别图片时崩溃","comments":8,"created_at":"2026-07-18T06:21:05Z"},{"number":4815,"title":"Android 内存泄露","comments":6,"created_at":"2026-08-27T11:30:26Z"},{"number":4811,"title":"Android API inference crash: SIGABRT in MNN::DeferBufferAllocator::apply() / MNN::Session::resize()","comments":5,"created_at":"2026-08-26T16:20:20Z"},{"number":5109,"title":"模型生成基准测试字符串失败","comments":2,"created_at":"2026-11-07T01:57:46Z"},{"number":5062,"title":"[Bug] Qwen3-TTS 0.6B: generation never emits EOS and decodes to digital silence for some texts (int4/int8/int3/fp16)","comments":2,"created_at":"2026-10-29T02:19:06Z"},{"number":5146,"title":"MNN Chat 加载/推理 DeepSeek-R1（7B 与 1.5B）时稳定闪退，两规格均复现","comments":1,"created_at":"2026-11-15T03:48:42Z"}],"discussions":[{"number":5068,"title":"3.6.2","comments":0,"category":"Announcements","created_at":"2026-10-29T09:38:23Z"},{"number":4662,"title":"3.6.1","comments":5,"category":"Announcements","created_at":"2026-07-23T02:13:33Z"},{"number":4546,"title":"3.6.0","comments":2,"category":"Announcements","created_at":"2026-06-16T07:58:06Z"},{"number":4355,"title":"MNN Supports Eagle3","comments":0,"category":"Technical Blog","created_at":"2026-04-07T04:13:21Z"},{"number":4354,"title":"MNN Model Support: Qwen3.5","comments":0,"category":"Technical Blog","created_at":"2026-04-07T03:39:57Z"},{"number":4353,"title":"How to Design High-Performance On-Device Tokenizers? MNN Refactoring Practice and Reflections","comments":0,"category":"Technical Blog","created_at":"2026-04-07T03:31:41Z"},{"number":4352,"title":"MNN Complex Task Skill Decomposition Experiment","comments":0,"category":"Technical Blog","created_at":"2026-04-07T03:25:58Z"},{"number":4351,"title":"TurboQuant: A Paper Crashes Storage Stock Prices? MNN Runs It on Mobile","comments":0,"category":"Technical Blog","created_at":"2026-04-07T03:17:49Z"}],"latest_release":{"name":"3.6.2","tag":"3.6.2","published_at":"2026-10-29T09:38:22Z","body":"# MNN 3.6.2 版本发布说明\r\n\r\n**发布日期**: 2026年10月\r\n\r\n📦 **安装**: `pip install mnn==3.6.2` ｜ [PyPI](https://pypi.org/project/mnn/3.6.2/)\r\n\r\n## 📌 版本概述\r\n\r\nMNN 3.6.2 聚焦 **端侧异构推理性能与投机解码加速**。本版本以 **Metal、Hexagon 和 DFlash** 为重点，持续提升 Qwen3 / Qwen3.5 在 CPU、GPU 与 NPU 上的推理效率。\r\n\r\n**典型性能：Qwen3.5-2B 在 iPad Pro M5 的 GPU 上通过 Metal 后端运行，Prefill 达到 3542.91 tok/s；启用 DFlash 后，Decode 从 111.89 提升至 233.98 tok/s，达到常规解码的 2.09 倍。**\r\n\r\n## 🚀 版本亮点\r\n\r\n*   **Metal / Apple GPU Neural Accelerators 优化**：通过 Metal Tensor API 利用 Apple GPU 内的神经网络加速单元，提升 Qwen3 / Qwen3.5 Dense 推理性能\r\n*   **Hexagon 推理优化**：提升 NPU 上的 Prefill 效率，减少 Decode 阶段的计算与调度开销\r\n*   **DFlash 投机解码加速**：加速草稿生成与验证，复用目标模型输出层权重，提高生成吞吐并减少内存占用\r\n*   **CPU / OpenCL 性能优化**：改进 OpenCL Attention 与融合解码，完善 CPU 的 SME2 检测与 SME2 / NEON 混合执行\r\n*   **LinearAttention 与 MoE 多后端优化**：提升 Qwen3.5 LinearAttention 在 Metal、Hexagon DSP、x86 与 RISC-V 上的执行效率，优化 MoE 专家计算\r\n*   **RISC-V / 进迭时空 K3 LLM 适配**：完善 RVV 与 IME2 支持，优化低比特 LLM 推理和动态输入处理\r\n\r\n## 📊 性能基准\r\n\r\n<img width=\"4080\" height=\"2448\" alt=\"performance_overview\" src=\"https://github.com/user-attachments/assets/897e778f-26dd-488e-8619-9854c81ca04d\" />\r\n\r\n*图：左侧为 iPad Pro M5 的 CPU / Metal GPU 性能，Qwen3.5-2B 额外展示 DFlash 投机解码；右侧为 OnePlus 13 的 CPU / OpenCL GPU / Hexagon NPU 性能。上排 Prefill，下排 Decode。*\r\n\r\n## ✨ 新功能与性能优化\r\n\r\n### Metal 后端\r\n\r\n*   **Apple GPU Neural Accelerators**：围绕 Metal Tensor API 优化矩阵计算、Attention 和 LinearAttention 的分块与数据访问，发挥 M5 等受支持 GPU 内神经网络加速单元的计算能力\r\n*   **Metal 低比特计算**：新增 W2 / W3 解码 GEMV 与 conv1x1 支持，优化 Q4 权重读取、分块计算和投机解码块处理\r\n*   **Metal Attention 与融合计算**：优化 prefill / decode Attention、RMSNorm、RoPE、融合 QKV / SwiGLU 投影及命令提交，减少中间计算与同步开销\r\n*   **Metal LinearAttention**：优化 gated delta rule 的融合计算，为 Tensor API 和 SIMD-group 矩阵计算提供分块 prefill 路径，并优化短输入和 decode 命令回放\r\n*   **Metal 上下文释放**：新增 context release API\r\n\r\n### Hexagon / NPU 后端\r\n\r\n*   **Hexagon 融合解码**：新增融合 INT4 GEMV、GatedRMSNorm 与 norm + RoPE 路径，减少解码阶段的算子调度\r\n*   **Hexagon DSP LinearAttention**：支持 Qwen3.5 gated delta rule 与 LFM2 short convolution，并引入分块 prefill 与 HMX 计算\r\n*   **Hexagon prefill 优化**：优化 INT4 反量化、HMX GEMM、分组因果 Attention、DMA 双缓冲和 VTCM 工作区，复用 prefill / decode 间的 KV 页面与动态内存池\r\n*   **Hexagon 模型与量化覆盖**：新增 Qwen3-VL 支持，完善对称 / 非对称分块量化、INT8 及 FP16 激活计算\r\n*   **Rockchip RKNN 后端**：新增 RKNN NPU 后端的初步支持\r\n*   **QNN 工具与算子**：新增 Deconvolution、中间 Tensor dump 和 VTCM 大小配置，支持 LLM / visual 模型导出至同一 QNN 目录\r\n\r\n### LLM / 推理与导出\r\n\r\n*   **MoE 专家计算优化**：将 Token 按专家归组，批量执行专家投影，融合 gate / up 与 SwiGLU；为 prefill 和 decode 提供对应的计算路径，覆盖 CPU / Metal / OpenCL / CUDA / Vulkan\r\n*   **Transformer 融合优化**：导出时融合投影层与 gated norm，并保留 split-LoRA 导出中的 FusedLinear 融合\r\n*   **DFlash 投机解码优化**：将特征投影、拼接与 argmax 放到设备端，增量维护草稿 KVCache，减少主机往返与重复计算；Metal 使用适合短验证块的 Q4 分块矩阵计算与 K 维切分\r\n*   **DFlash 权重复用**：草稿模型共享目标模型的 lm_head，减少磁盘与运行时的重复权重\r\n\r\n> **升级提示：旧版 DFlash 草稿模型需重新导出。**\r\n\r\n*   **QAT checkpoint 导出**：新增 `--qat`，读取 `qat_config.json` 与 safetensors，保留 scale / zero_point，支持 2 / 3 / 4 / 8-bit 权重\r\n*   **可选文本 Token 压缩**：新增 `MNN_LLM_TOKEN_COMPRESS` 编译选项和 `token_compress` 配置，按 Token 预算选取文本句子；默认关闭，支持纯文本请求及多模态模型的纯文本输入\r\n*   **OpenCL 前缀 KVCache 文件**：新增 `setPrefixCacheFile` 支持\r\n*   **Attention mask 统一**：统一 mask 构造及各后端的处理语义\r\n\r\n### CPU 后端\r\n\r\n*   **SME2 计算单元检测**：识别可用计算单元及共享分组，为任务划分提供准确的硬件信息\r\n*   **SME2 / NEON 混合优化**：依据计算单元数量分配矩阵任务，优化 Attention、INT8 和 KleidiAI 混合执行，减少线程调度开销\r\n*   **Arm Attention / RoPE / LayerNorm**：优化 FlashAttention prefill / decode，新增 Arm82 FP16 RoPE，优化 packed FP16 C4 LayerNorm\r\n*   **RISC-V RVV 算子扩展**：新增或优化 LinearAttention、Attention mask、KVCache 量化、GEMV、矩阵计算、卷积 / 反卷积、采样、Scale / PReLU 和 INT8 算子\r\n*   **进迭时空 K3 / IME2**：优化低比特矩阵计算与激活打包，在动态 prefill 中保留执行对象并复用权重预打包结果，减少输入长度变化时的重复准备开销。采用 **8 线程、W4B64** 配置，Qwen3.5-2B 实测 Prefill（PP512）达到 **85.60 tok/s**，Decode（TG128）达到 **17.68 tok/s**\r\n*   **卷积与图像算子**：优化反卷积输出复用、2×2 反卷积后处理及双线性采样\r\n\r\n### 其他 GPU 后端\r\n\r\n*   **OpenCL FlashAttention 与调优**：重构 prefill / decode kernel，完善 mask、尾块、KV 布局和长输入处理；优化异步准备、运行时调优与 kernel 程序拆分\r\n*   **OpenCL 融合解码**：融合 C4 gated / binary RMSNorm 和 FusedLinear GEMV，优化 decode RoPE，减少 QCOM record replay 同步开销\r\n*   **Vulkan cooperative matrix**：优化 INT4 矩阵计","is_prerelease":false},"readme":true,"readme_zh":true,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}