{"full_name":"ggml-org/llama.cpp","owner":"ggml-org","name":"llama.cpp","shard":"1","url":"https://github.com/ggml-org/llama.cpp","description":"LLM inference in C/C++","homepage":"https://llama.app","topics":["ggml"],"tag_pages":[],"language":"C++","license":"MIT","avatar":"https://avatars.githubusercontent.com/u/134263123?v=4","created_at":"2023-03-10T18:58:00Z","pushed_at":"2026-11-15T19:28:56Z","metrics":{"stars":135174,"forks":25641,"watchers":926,"issues":10872,"prs":17262,"releases":8170,"issues_open":705,"issues_closed":10167,"prs_open":1911,"prs_closed":3691,"prs_merged":11660,"discussions":3890},"growth":{"day":{"stars":134,"forks":38,"base_day":"2026-11-15"},"week":{"stars":1381,"forks":362,"base_day":"2026-11-09"}},"history":{"dates":["2026-11-09","2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[133793,133886,134071,134282,134774,134862,135040,135174],"forks":[25279,25324,25400,25473,25531,25566,25603,25641],"watchers":[874,874,907,925,925,926,923,926],"issues_open":[null,null,712,703,708,711,706,705],"prs_open":[null,null,1883,1864,1875,1894,1914,1911]},"boards":{"daily":178,"weekly":130,"daily_forks":107,"daily_watchers":59,"total":86},"same_owner":[{"full_name":"ggml-org/Llama-macOS","name":"Llama-macOS","shard":"1","boards":{"daily_watchers":191,"debut":24}},{"full_name":"ggml-org/ggml","name":"ggml","shard":"1","boards":{"daily":92,"daily_rate":115,"daily_forks":25}},{"full_name":"ggml-org/whisper.cpp","name":"whisper.cpp","shard":"1","boards":{}}],"issues":[{"number":10528,"title":"Misc. bug: Inconsistent Vulkan segfault","comments":83,"created_at":"2024-11-26T19:54:03Z"},{"number":10982,"title":"Research: Performance differences between Metal (macOS) and Vulkan (Linux)","comments":58,"created_at":"2024-12-26T11:12:21Z"},{"number":14909,"title":"Feature Request: Implement missing ops from backends","comments":56,"created_at":"2025-07-28T03:39:11Z"},{"number":2164,"title":"mpi : attempt inference of 65B LLaMA on a cluster of Raspberry Pis","comments":54,"created_at":"2023-07-10T16:12:22Z"},{"number":11970,"title":"Misc. bug: The KV cache is sometimes truncated incorrectly when making v1/chat/completions API calls","comments":50,"created_at":"2025-02-20T11:20:01Z"},{"number":24066,"title":"Eval bug: Vulkan: performance drop in recent builds","comments":49,"created_at":"2026-06-03T10:06:36Z"},{"number":25618,"title":"Eval bug: Speculative decoding (draft-mtp / draft-dspark): greedy output diverges from vanilla on quantized targets","comments":43,"created_at":"2026-07-13T11:55:59Z"},{"number":14702,"title":"Misc. bug: OpenAI API v1/responses llama-server","comments":41,"created_at":"2025-07-15T21:10:07Z"}],"discussions":[{"number":32298,"title":"RFC: Q2_0C/Q4_0C weight quantization with per-channel scale and offset","comments":3,"category":"Ideas","created_at":"2026-11-15T00:23:21Z"},{"number":32285,"title":"Opt-in sparse-backed KV cache allocation (Vulkan / Linux CPU) to avoid upfront VRAM reservation","comments":0,"category":"Ideas","created_at":"2026-11-14T17:46:13Z"},{"number":32234,"title":"Capping CPU clock on a hot laptop: same MoE token speed, 16 °C cooler (GPU/CPU expert split)","comments":2,"category":"Show and tell","created_at":"2026-11-13T22:19:58Z"},{"number":32226,"title":"Quantized KV + --kv-unified is much slower than f16 KV on long context, and a PR proposal for its improvement","comments":1,"category":"General","created_at":"2026-11-13T18:46:36Z"},{"number":32211,"title":"Fusion: a cloud or local planner driving llama-server parallel slots as a pool of agent workers","comments":1,"category":"Show and tell","created_at":"2026-11-13T14:54:27Z"},{"number":32180,"title":"Reduce Grammar Reliance for Tool Calling","comments":1,"category":"Ideas","created_at":"2026-11-13T07:08:58Z"},{"number":32171,"title":"Lossless Speculative Decoding for Looped Models (Ouro Family): Draft with 1 Loop, no Draft Model","comments":1,"category":"Ideas","created_at":"2026-11-12T23:58:41Z"},{"number":32165,"title":"Ryzen 7 255, Radeon 780M benchmarking Vulkan versus ROCm (7.2)","comments":0,"category":"General","created_at":"2026-11-12T21:37:18Z"}],"latest_release":{"name":"v1.0.0","tag":"v1.0.0","published_at":"2026-11-13T10:20:19Z","body":"<img width=\"1600\" height=\"900\" alt=\"llama-cpp-v1\" src=\"https://github.com/user-attachments/assets/ed94ed55-ebe2-4aee-bdfc-81bdaa3a3f59\" />\r\n\r\nBlog post: https://huggingface.co/blog/ggml-org/llama-cpp-v1\r\n\r\n## Overview\r\n\r\nllama.cpp v1.0.0 removes the deprecated public API and promotes the NextN/speculative API to `llama.h`, adds OCP FP8 KV caches, a host-memory MoE expert cache with a configurable CPU share, and loading of external PLE shards. It adds three text architectures (Kimi-K3.1-Pro, Ripple and xing4_0), drama-3 TTS, and MageVL-Cowork and Kimi-K3.1 vision, hardens `llama-server` (SSRF, CORS, DNS rebinding, MCP tool cancellation) and adds a `/v1/audio/speech` endpoint, ships a redesigned web UI with chat groups and a markdown editor, and updates `ggml` to v1.0.0.\r\n\r\n### API changes\r\n\r\n- llama.h: remove `llama_set_warmup`, 35 deprecated functions, `defrag_thold` and `--defrag-thold` ([#30565](https://github.com/ggml-org/llama.cpp/pull/30565))\r\n- llama.h: promote the NextN API from `llama-ext.h` and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src` ([#30565](https://github.com/ggml-org/llama.cpp/pull/30565))\r\n- llama.h: add `moe_cache_size` and `moe_cache_cpu_share` to `llama_context_params` ([#30966](https://github.com/ggml-org/llama.cpp/pull/30966))\r\n- mtmd.h: add `MTMD_GEN_AUDIO_TYPE_DRAMA3` and `temp` in `mtmd_helper_gen_audio_inp` ([#31613](https://github.com/ggml-org/llama.cpp/pull/31613), [#31655](https://github.com/ggml-org/llama.cpp/pull/31655))\r\n- mtmd-helper.h: add `mtmd_helper_resampler` for streaming raw PCM at the model sample rate ([#31964](https://github.com/ggml-org/llama.cpp/pull/31964))\r\n- server: add the OpenAI-compatible `/v1/audio/speech` endpoint and a `temperature` option for `/tts` ([#31763](https://github.com/ggml-org/llama.cpp/pull/31763), [#31655](https://github.com/ggml-org/llama.cpp/pull/31655))\r\n\r\n### New models\r\n\r\n- Kimi-K3.1-Pro: K3 with the full-attention MLA layers replaced by DSA ([#31779](https://github.com/ggml-org/llama.cpp/pull/31779))\r\n- Ripple-65B-Preview (`ricursive`): new dense 65B architecture, no new ops ([#31692](https://github.com/ggml-org/llama.cpp/pull/31692))\r\n- xing4_0: mHC, MLA, MoE and MTP ([#31589](https://github.com/ggml-org/llama.cpp/pull/31589))\r\n- drama-3 TTS (fishaudio) ([#31613](https://github.com/ggml-org/llama.cpp/pull/31613))\r\n\r\n### Core changes\r\n\r\n- FP8 KV caches (`-ctk/-ctv f8_e4m3`) with optional static per-tensor scales ([#28392](https://github.com/ggml-org/llama.cpp/pull/28392))\r\n- MoE cache: CPU share for misses, LRFU/ClassLRU replacement, no CPU wait on cached experts ([#30966](https://github.com/ggml-org/llama.cpp/pull/30966), [#31233](https://github.com/ggml-org/llama.cpp/pull/31233), [#31888](https://github.com/ggml-org/llama.cpp/pull/31888))\r\n- external PLE/engram loading from separate GGUF shards via `--shard`/`--ple` ([#31185](https://github.com/ggml-org/llama.cpp/pull/31185))\r\n- Windows unbuffered model load, mirroring the Linux DirectIO path ([#26014](https://github.com/ggml-org/llama.cpp/pull/26014))\r\n- reuse the backend scheduler and graph results across reserves, and skip unused graph metadata pages ([#32009](https://github.com/ggml-org/llama.cpp/pull/32009), [#32089](https://github.com/ggml-org/llama.cpp/pull/32089), [#32043](https://github.com/ggml-org/llama.cpp/pull/32043))\r\n- mark every sampling chain as a graph output to prevent output aliasing ([#32015](https://github.com/ggml-org/llama.cpp/pull/32015))\r\n- remove the non-fused GDN paths and the ubatch-dependent graph node budget ([#32078](https://github.com/ggml-org/llama.cpp/pull/32078))\r\n- store the MTP conv snapshots of every recurrent model in one `set_rows` ([#31660](https://github.com/ggml-org/llama.cpp/pull/31660))\r\n- per-model `build_rope` with a KV-shifting fix and tests ([#31683](https://github.com/ggml-org/llama.cpp/pull/31683)); the model is a member of `llm_graph_context` ([#31733](https://github.com/ggml-org/llama.cpp/pull/31733))\r\n- fix on-device state copy s","is_prerelease":false},"readme":true,"readme_zh":false,"news":[{"title":"Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI","url":"https://github.com/ggml-org/llama.cpp/discussions/19759","hn":"https://news.ycombinator.com/item?id=47088037","source":"Hacker News","lang":"en","date":"2026-02-20","points":839,"comments":223},{"title":"Vision Now Available in Llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md","hn":"https://news.ycombinator.com/item?id=43943047","source":"Hacker News","lang":"en","date":"2025-05-10","points":550,"comments":104},{"title":"Mistral Integration Improved in Llama.cpp","url":"https://github.com/ggml-org/llama.cpp/pull/14737","hn":"https://news.ycombinator.com/item?id=44862583","source":"Hacker News","lang":"en","date":"2025-08-11","points":95,"comments":15},{"title":"Llama.cpp: Add GPT-OSS","url":"https://github.com/ggml-org/llama.cpp/pull/15091","hn":"https://news.ycombinator.com/item?id=44799218","source":"Hacker News","lang":"en","date":"2025-08-05","points":35,"comments":0},{"title":"LLama.cpp Got Screwd","url":"https://github.com/ggml-org/llama.cpp/discussions/25482","hn":"https://news.ycombinator.com/item?id=48844268","source":"Hacker News","lang":"en","date":"2026-07-09","points":11,"comments":5},{"title":"Tell HN: Llamacpp now supports unified system RAM offloading on Linux","url":"https://news.ycombinator.com/item?id=47510953","hn":"https://news.ycombinator.com/item?id=47510953","source":"Hacker News","lang":"en","date":"2026-03-24","points":6,"comments":0},{"title":"Yet another reminder why you should not use Ollama","url":"https://github.com/ggml-org/llama.cpp/pull/19324","hn":"https://news.ycombinator.com/item?id=46895014","source":"Hacker News","lang":"en","date":"2026-02-05","points":6,"comments":5},{"title":"Is GitHub Releases Down?","url":"https://news.ycombinator.com/item?id=44303056","hn":"https://news.ycombinator.com/item?id=44303056","source":"Hacker News","lang":"en","date":"2025-06-17","points":5,"comments":3},{"title":"What's easier in llama.cpp v0.8.0? Finding models, long conversations, and things to watch out for after upgrading","url":"https://note.com/ai_gyaru_lab/n/n98a992874ad4?hl=en","source":"note","lang":"en","date":"2026-10-29"},{"title":"Build llama.cpp with DFlash 2 support (PR #27342): git clone","url":"https://x.com/analogalok/status/2092181751363936543","source":"x","lang":"en","date":"2026-08-25"},{"title":"Accelerate llama.cpp on AMD EPYC: ZenDNN Delivers Up to 4.5x the Prompt Processing","url":"https://www.amd.com/en/developer/resources/technical-articles/2026/llama-cpp-on-amd-epyc.html","source":"amd","lang":"en","date":"2026-08-18"},{"title":"I ditched LM Studio for llama.cpp, and my local LLM doesn't feel like a downgrade anymore","url":"https://www.xda-developers.com/ditched-lm-studio-for-llama-cpp-and-local-llm-doesnt-feel-like-downgrade-anymore/","source":"XDA Developers","lang":"en","date":"2026-05-23"},{"title":"Free software 'llama.cpp' that can run various AI models locally supports multimodal input and enables image explanations, etc.","url":"https://gigazine.net/gsc_news/en/20250512-llama-cpp--multimodal-vision-support/","source":"GIGAZINE","lang":"en","date":"2025-05-12"},{"title":"Intel announces that 'Run llama.cpp Portable Zip on Intel GPU with IPEX-LLM', which allows various AIs to run on a local Windows PC, also supports DeepSeek","url":"https://gigazine.net/gsc_news/en/20250309-run-llama-portable-zip","source":"GIGAZINE","lang":"en","date":"2025-03-09"},{"title":"``llama.cpp'', which made Meta's large-scale language model ``LLaMA'' executable on a Mac with M1, can be executed with less than 6 GB of memory usage by updating","url":"https://gigazine.net/gsc_news/en/20230403-llama-cpp-ram","source":"GIGAZINE","lang":"en","date":"2023-04-03"},{"title":"BaseRT: 在 Apple 芯片上通过原生 Metal 实现同类最佳的 LLM 推理","url":"https://news.google.com/rss/articles/CBMiVEFVX3lxTFB2R0ZVQldrYkVVSVdoN21iQUZHYnRJeGpBVG1IVlNrbTF0OGtzV0R0LVQ0QVJQRHhEbkZFdHhBM3Y5YnJkWWI0aHg3TFY2TTBQY0hYaw?oc=5","source":"alphaXiv","lang":"zh","date":"2026-09-02"},{"title":"Run Muse Glimmer for Local Vibe Coding with llama.cpp, DFlash, and Pi","url":"https://news.google.com/rss/articles/CBMimAFBVV95cUxOaEowQkR4YTljaVN1bUk1Z1BNTlRWNzVPY2xTa21Wc3d6S21XVVdyWjdYMDFBdGdhTVlONHVGc0x3TGhUM2M3LUlaWG8wVmdrenJjVk4tWDZDRGZDYUFfajRISF9ncjFqLVpBODl5YWxTbWN1UFVpclBsTmlLOVozaWVmVy1mM2xUN0pQVkNoWE5vclFabmdGcg?oc=5","source":"KDnuggets","lang":"en","date":"2026-08-21"},{"title":"Windows本地AI升级，llama.cpp支持多后端，一键跑GGUF无审查模型","url":"https://news.google.com/rss/articles/CBMiW0FVX3lxTE5tSTkyR0hoYXVZME5VcHV0cXpHWnRLS1NqR2Q0cGVITDZYSGx0VTh5d21VdHRUb2ozRUxkM09OTThvWm9SUUVMUktDSHVqMEd5dnEtd2hNNDdrdTg?oc=5","source":"blog.upx8.com","lang":"zh","date":"2026-04-08"},{"title":"Running LLMs on Raspberry Pi and Edge Devices: A Practical Guide","url":"https://news.google.com/rss/articles/CBMiXkFVX3lxTFBpVnVtRTVaYkV1TEFuNFV5aW1LaV9kYmFhUlBnS3RudHcxOHA4ekc1d1lhTG9QVXpFdFVSaWUyU2s0MGZqZ09CbFJLaEMzMTd5NjAxblF6T0lPaUl2TUE?oc=5","source":"SitePoint","lang":"en","date":"2026-02-25"},{"title":"GGML Joins Hugging Face: What This Means for Local Model Optimization","url":"https://news.google.com/rss/articles/CBMimwFBVV95cUxNc0VZdEE0SnlBTnF6bERlVjNXWV9mUW1Va2pVODU3X3VsN0JkdjQ1bnRsczkySnl0MkxHY0J2SE1tS21GLUVsMnFNcG85TWRhbTVPbHkwMktIcU1HZFhiSU5RR2lNOUxaNHhwSFpaeFUzMXBpTjh3cDh4QldrQUI4Wm43ZXN3NDVBT2NMZlllVUF0SUJzTEpkZkg3NA?oc=5","source":"SitePoint","lang":"en","date":"2026-02-22"},{"title":"How to run LLMs on PC at home using Llama.cpp","url":"https://news.google.com/rss/articles/CBMioAFBVV95cUxPbHh2dFU3bDh6d0NoOW5rYkFnblhFVHhNMl8zR3FyUjU1dDlGcTdZWml5b1R3dDUtdi1EZXB1Y2x1MF81M3N2YmZ3Q1JTWjJ5eUxVMmFpZVpXandtYUNybURTbXpsS3o5SV9pS3ZCU09vMmJ2MVhXX2JjMWVqMUNxcWhTbGVweHdTdHZvc0c4OGJTbklVaUlCRTFSR04yTFhP?oc=5","source":"The Register","lang":"en","date":"2025-08-24"},{"title":"在llama.cpp中为高通Adreno GPU引入全新的OpenCLTM GPU后端","url":"https://news.google.com/rss/articles/CBMiXkFVX3lxTE1DcmJWMGplVXdCVDA0b0xSVG1iQWY3M1ZIRzN1bkc3NDUzY2QyTVhOVWV5eWVUdEZCcWJtX0NlX1ZhaWpDeGZBekhZYzNOcXBPMTBqc0Rsc2xuZlJDelE?oc=5","source":"CSDN","lang":"zh","date":"2024-12-11"},{"title":"Optimizing llama.cpp AI Inference with CUDA Graphs","url":"https://news.google.com/rss/articles/CBMijgFBVV95cUxOUldOYmNTRXlDM01naE4yVnNuS01Hc3B0VS04SE5HaXJyUTlQREhtLWg4VVE0Um9aWHdfMmZtNUo0UG8xWkNBNi1BX2dXYThBYUlVLU5zREgxdHoyaXhHWG9MSkhyZ3BoSjdkU294WTFDSkt2UU9tTmNMc1BnM1pvVlZuaXBWZ2pWWEI3QThn?oc=5","source":"NVIDIA Developer","lang":"en","date":"2024-08-07"},{"title":"研究完llama.cpp，我发现手机跑大模型竟这么简单","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTE5lUUdFeUo2bHZWMDBrTngyN1QydUo4eFpIcExBWlNOTGllT2Z3TXJsT3lQbzRoM3I1UDF0NUxHaUd3bzBoQ0Zfbw?oc=5","source":"品玩","lang":"zh","date":"2023-08-17"},{"title":"Llama 2本地运行指南和工具Ollama，已4k星","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTFBxVkJDbi1NemxMWHdOZ2hQazAtNS1kNkQ0TGVjbWZXcWRXT0ZGeHd4aUhoeUhMWWlHN2xmbGVZeTJjOWZobWcycQ?oc=5","source":"智源社区","lang":"zh","date":"2023-08-03"},{"title":"3w＋星标项目大佬创业：树莓派就可运行大模型，获GitHub前CEO投资","url":"https://news.google.com/rss/articles/CBMiVkFVX3lxTE5hVnJ5TkFBYTBRQ2UyVWl6eEJrVm55Qmw4emVsRUlRVTdkMi1tb1hLQkpXbGRFUnVQSlozSmxIRGxmaGwtMEU2UGZ1VDNxT05jOHJTSkln?oc=5","source":"QQ News","lang":"zh","date":"2023-06-25"}],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}